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INTRODUCTION 

Purpose and Approach 

This resource paper offers a guide for a dual purpose evaluation plan 
that can provide formative Information for local program managers and 
simultaneously serve the accountability and reporting needs of a state 
legislature, district office, or other policymaking body. While the 
primary audience for this paper 1s expected to be those Interested 1n 
educational evaluation, and the examples provided will be from this field, 
we hope that this paper will also prove Interesting to those Involved 1n 
evaluation of non-educational programs. The approach presented here 
reflects findings from several years of CSE research on a formative 
orientation to evaluation, on qualitative methodology, and on strategies 
for maximizing the utilization of evaluation efforts. 

There are three principles that guide our approach. First and 
probably most important, we believe that evaluation should serve the needs 
of a multiplicity of users. Teachers, counselors, and program directors 
continually want evaluation Information to refine and improve the program 
with which they work. In addition, policymakers for such programs (be they 
at the district, state or federal level) also need information to guide 
their policy- and decisionmaking. It is clear that these two demands— the 
top-down demand for broad-level accountability (to improve management and 
to elevate standards of excellence) and the bottom-up demand for adaptive, 
sensitive infor atlon to be useful at the local level— push in different 
and not totally compatible directions. Because it has usually been assumed 
that these different decisions require different types cf data, separate 
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evaluations have usually been conducted to meet these two needs. If we 
consider the result when these evaluations are added to all the other data 
collection efforts that occur (e.g., for auditing, accreditation, diagnosis 
and prescription), we can begin to comprehend why many students, teacherr 
and program administrators feel overwhelmed by evaluation. Furthermore, 
much of the data collected is of limited utility for them because 1t often 
fails to reflect the actual school context and curriculum. They view 
evaluation as burdensome and Intrusive and come to resent the top-down 
demand for accountability, seeing it as more of a liability than an asset. 

Other problems arise when policymakers and program staff have no 
common basis for their separate Inferences about policies and practices, 
irst, the general intent of educational policy formation 1s to Improve the 
quality of educational services and to help students attain the highest 
levels of compentency in school subjects. To accomplish this goal, the 
policies must be translated Into practices that are compatible with the 
views, needs and capabilities of teachers and students. Second, local 
programs alone do not necessarily have sufficient resources to solve their 
problems. The solutions may require initiative, directions, resources and 
action at those higher levels charged with responsibility for governance, 
resource allocation and policy formation. In both cases there 1s high 
potential for slippage when the Information used to assess quality and 
formulate policy functions Independently from that used to actually teach 
students. 

In summary, the current system of Independent evaluations appears both 
uneconomical of time and effort and Ineffective 1n accomplishing the goals 



9 



3 



of either end of the educational continuum. A partial solution to this 
problem, presented here, 1s a single evaluation effort that uses a common 
Information base to serve both policy and program needs simultaneously. 

The second principle guiding our development of this paper 1s that 
evaluation, testing and standard setting are endeavors which are partly 
technical, partly political, and partly social. Technical expertise 1s 
essential 1n measurement development and analysis, to ensure valid and 
reliable use of results. Social understanding 1s essential to ensure 
fairness and utility. Similarly, evaluation questions arise out of 
people's Information requirements, while the design and Interpretation of 
evaluations depend on technical competence. The definition of standards 
depends on values and consensus; the measurement of their attainment 
Involves technical considerations. Thus 1t seems crucial that a useful 
evaluation plan will draw on all these areas of expertise and Involve 
professional evaluators as well as the range of users at both levels of the 
system. 

The final principle 1s that effective evaluation plans are 
characterized by several key features: 

(a) evaluators who are personally committed to the use of results and 
who have political sensitivity, credibility, and rapport with users; 

(b) users who are also committed to Implementing the results of the 
valuation, open to change, and Involved 1n the process so that 1t reflects 
thalr leadership, expectations, and perceptions of needs and risks; 

(c) a setting 1n which representatives of the major stakeholders agree 
on the focus of the evaluation and the kinds of Information 1t should 
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produce, and in which local program personnel have the autonomy to act upon 
the evaluation's findings; and 

(d) the evaluation procedure 1s deci sion-b>sed, allows purposeful 
sharing of ideas, and provides timely, specific and relevant "information 
(Burry, Alkin & Ruskus, 1985). 
Potential Users 

The evaluation process described in this paper reflects the need for a 
systematized, short term, qualitative evaluation plan that addresses both 
the desire of local programs for formative, program planning and 
improvement-oriented information as well as the needs of policymakers for 
standardized information across multiple sites for the purpose of summative 
decisionmaking, often officially mandated. This evaluation process is 
intended to be appropriate in a setting characterized by multiple program 
sites with similar missions (although a single program site could also 
make use of this plan) overseen by a separate policymaking body. Such 
settings might include a foundation or community agency with several 
programs to oversee or the typical educational settings at every age level 
and level of organization. Specific examples include: (a) a state office 
overseeing special admissions programs for minority students at each campus 
of a statewide community college system, (b) a county agency overseeing 
child care information and referal systems in several cities, and (c) a 
school district office overseeing bilingual programs in the elementary 
schools in the district. 
Structure of the Resource Paper 

The dual purpose evaluation model presented here is described in eight 
steps: 
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1. Identify evaluation users and their needs. 

2. Form task force. 

3. Reach consensu on information needs. 

4. Specify common Information base and develop measures, 

5. Develop procedures. 

6. Pilot and refine measures and procedures. 

7. Collect data. 

8. Prepare reports for policymakers and program personnel. 

Where appropriate, specific examples are provided from our experiences with 
several evaluation projects. The report concludes with a summary of 
critical concerns. 

DUAL PURPOSE EVALUATION PLAN: A MODEL PROCESS 
Step 1. Identify Evaluation Users & Their Needs 

This evaluation process 1s designed to meet the nee H < ^f two levels of 
users: policymakers and program personnel. Each of these has somewhat 
different needs. The purpose of this first step is to identify the 
specific policymakers (e.g. members of the school board and state 
legislators) who arc responsible for the programs and to identify those 
individuals Involved on-site with the actual implementation of a program 
(e.g. faculty, staff, and local administrators). 

The policymakers usually need su,nmat1ve Indicators reflecting program 
goals and outcomes across all sites at which a particular type of program 
is Implemented. These Indicators will focus on broad patterns and trends 
emerging across program sites rather than on the program as Implemented at 
an individual site. To obtain this information policymakers need a 
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comparable conceptual scheme and measurement base for each site. Where 
many sites are Involved, they may employ an evaluation cycle 1n which some 
percent of the total number of sites 1s evaluated 1n * given year and each 
site participates only once every several years. T\as 1n a given year, the 
data 1s collected from only a sample of the ^hole system and 1s aggregated 
so as to provide a view of the system rather than of the Individual program 
sites monitored that year. 

The program personnel, on the other hand, need specific Information to 
pinpoint the strengths and weaknesses of their Individual Implementation of 
the program 1n order to make Improvements, They *ant to assess the extent 
to which the Intended program has actually been Implemented and the 
outcomes that have been achieved, both Intended and unintended. 

Step 2. Form Task Force 

Including representatives of all major users of the evaluation (at 
both the program and policymaking levels) as members of a task force to 
oversee the evaluation facilitates both the process and the utilization of 
results. Many ?n evaluation has been undermined by evaluation us*rs 
who failed to understand the Intent, felt threatened by the potential for 
change, had political agendas at variance with the goals of the evaluation, 
were used to pro forma evaluations 1n which results were never utilized, or 
were faced with Information that could not be clearly Interpretted for 
action. Involving users early 1n the process helps avoid theje problems. 
It provides valuable Input at the point when 1t is most useful rather than 
later, thereby causing duplication of effort. It also gW those 
participants a sense of ownership of the evaluation which makes 1t much 
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more likely that the results will actually be used for the intended 
purposes. 

The responsibilities of a task force are: 

(a) to define the dual focus of the evaluation and reach consensus on 
what information 1s needed 1n the evaluation, 

(b) precisely specify the Information base and select measures to 
cullect the Information, 

(c) plan the data collection and analysis procedures, 

(d) pilot test and refine the measures, materials, and procedures, 

(e) oversee collection *f the information, and 

(f) prepare reports for policymakers and for program personnel. 

In some cases, the task force may preside over a single evaluation. 
In other circumstances, the task force may be an ongoing body (whose 
members may change over time) that oversees all two-tiered evaluations of 
the target programs. In either case, the- task force will consist of the 
general task force members, tlie task force director, and one or more 
evaluation experts. 

What kinds of backgroL .id skills are sought in task force 
members? First, they need to represent the major stakeholders in the 
evaluation: policymakers rid program personnel. Where programs serve a 
number of special groups or provide several services (e.g. a program for 
the handlcapoed may serve those with learning disabilities and those with 
visual Impairments in the same program"), 1t is Important to include task 
force members with expertise 1n each of the subgroups or subprograms. 

It is Important that the task force not be too large to reach 
agreement on crucial decisions and not be too small to Include the full 



14 



to effect recommended changes (Alkin, 1985). People without evaluation 
experience may be used to operating from Intuition rather than from data, 
so the duties of the task force may be new and somewhat uncomfortable to 
them at first. Prior experience 1n evaluation or research 1s useful 1n 
providing appreciation for the necessary technical aspects of the measures 
and for the feasibility of the data collection plan. In addition, 
recommendations on these matters from external evaluators are more likely 
to be appreciated and Incorporated without undue delay or resistance. On 
the other hand, previous experience with poorly conceived or conducted 
evaluation efforts or those 1n which results were not used may have led 
members to distrust evaluators and to expect that the results of the 
evaluation will never really be utilized, especially for local program 
improvement. If several members fall 1n this category, a good deal of time 
may be needed to change their attitude. 

Who should direct the task force? Considering the dual focus of the 
evaluation, the director murt have several characteristics — social, 
political, managerial, professional and technical — that will help ensure 
the success of the endeavor. It is Important that all task force members, 
especially representatives of proqram personnel (who commonly feel that 
evaluation is a top-down Imposition), view the director as "one of us." A 
site director of one of the programs, a peer of the other members, may fill 
this need well if he or she also possesses other necessary traits. Since 
the members will undoubtedly have divergent opinions on most Issues yet be 
required to reach consensus, it 1s imperative that the director have 
excellent Interpersonal and communication skills and have power and 
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political acumen. The director's effectiveness 1n helping the group reach 
consensus may reflect his or her ability to use the group dynamics to 
advantage, for example utilizing peer pressure on members who may create 
problems or stall group progress. The sheer size of the task also requires 
that the director have strong organization and management skills. 

Due to the nature of the task, the task force director must not only 
have a strong background 1n evaluation but also be personally committed to 
seeing the results put to use. Research and experience suggest that 
evaluations are more effective when users are included in all aspects of 
the evaluation (Burry, Alkin, & Ruskus, 1985). This may necessitate that 
the director tactfully educate some task force members and local program 
personnel about evaluation or measurement concerns. In order to guide the 
task force well, the director also should have a good sense about how much 
Information 1s both necessary and feasible to collect. 

During at least the Initial evaluation, if not the entire life of the 
task force, the expertise, guidance and encouragement of an external 
evaluation expert 1s likely to be needed. The evaluator must have 
credibility In the eyes of both the program staff and policymakers. Since 
the evaluator's role is to provide technical expertise and assist the task 
force, the evaluator must share many of the director's qualities. As with 
the director, 1t 1s Important that the evaluator be personally commlted to 
seeing the results put to use. The evaluation 1s more likely to be 
successful 1f the evaluator enjoys a good rapport with the task force 
members and adopts a collaborative role, 1n which he or she views the users 
as colleagues who can help guide the evaluation and who have legitimate 
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questions * are entitled to the evaluator's attention. The evaluator 
must be sensitive to the program's political dynamics, background, power or 
prestige. Group facilitation skills will undoubtedly be useful in 
resolving conflicts and negotiating Issues. The evaluator must have ample 
experience and understanding of the technical requirements of the sorts of 
measures to be used as well as a realistic expectation of feasible 
measurement procedures 1n the given setting. As the task force and ics 
mission attain comfcrtable age and experience, the need for an external 
evaluator may recede. 

Step 3, Reach Consensus on Information Needs 
The first duty of the task force members 1s to define the purpose of 
the evaluation and set the ground rules. Beyond agreeing that the 
evaluation should provide Information to guide both policymaking and 
program improvement, the task force must deal with what balance between 
these two needs is to be their goal. Are both to be emphasized equally, or 
should one be more heavi iy weighted? How is the Information to be used? 
Will it be tied to an audit or accreditation? Is it necessary or desirable 
for site-specific Information to be kept anonymous at the policymaking 
level" 

The second duty of the task force 1s to reach consensus on what 
specific kinuS of Information need to be collected. A11 task force members 
should participate in this process, drawing on Input from others at the 
policymaking office and the Individual sites. Users 1 Information needs may 
be stated 1n terms of questions to be answered or Issues to be addressed. 
Clarifying why users need the given Information will help the group set 
priorities and be certain that their intentions will really meet the need 
precisely — not just meet 1t "almost." 

ERIC 18 
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Consensus on the general kind of Information that needs to be obtained 
across all sites will provide a uniform Information base for the two 
evaluation purposes. This uniform information base will be referred to 
here as the "core" Information, Indicating that 1t contains the basic 
Information that 1s required from each program site. In addition, the task 
force may decide to allow Individual sites to supplement the core 
Information with additional s1 te-spec1f1c Information to be used In program 
Improvement. If this 1s the case, the task force needs to reach agreement 
on the type and amount of Information each site can collect to ensure that 
1t will not jeopardize the success of evaluating the core Issues. 

Consensus on Information needs 1s necessary because 1t guides all 
future activities of the task force and data collection efforts, 
particularly development of the specific measurement Instruments used at 
every program site; hence, real, not just rhetorical, agreement by all task 
force members 1s Imperative for the evaluation to be successful. 

One strategy for reaching consensus 1s for the task force director to 
present a fairly comprehensive 11st of tentative questions or Issues to 
which task force members may add others. Members then rank order the Issues 
by Importance. Alternatively, each issue could be rated according to Its 
degree of Importance. Results can be tallied and further discussed by the 
group until wholehearted agreement 1s reached. A good rule of thumb 1n 
reviewing all proposed Issues or questions, either for the core or the 
optional portion of the evaluation, 1s: How, specifically, will this 
information be used to Improve the program or make policy? Will this piece 
of Information really help us make our decisions? The tendency 1s to say, 
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"That sounds Important; of course we need to know that, 14 without seriousl> 
questioning how the Information will be summarized and translated into 
proposals for action. 

Continued agreement across time may require periodic discussion and 
particularly clarification, 1f new members are added to the task force or 
new concerns are encountered. Significant changes 1n the purpose, balance, 
or Issues addressed that occurs at any later point 1n the process will 
undoubtedly result 1n wasted time and effort. Hence, 1t 1s Important to 
allow sufficient time during this itep for the task force to reach a 
consensus with which everyone 1s satisfied before proceeding. 

Typically, the core Information needs Include: 

a. Who is Involved? — a description of program participants 

b. What does the program consist of?— a description of program 
services and/or Instruction (which may compare what the program actually is 
with what 1t 1s Intended to be) 

c. What are the program's effects?— outcome and experiential data for 
all participants — students, staff and faculty — Including performance and 
attitude data on both intended and unintended outcomes 

d. What are the program's strengths and weaknesses? — conclusions and 
recommendations based on the other information. 

Step 4. Specify Information Base and Develop Measures 
After the information needs have been defined by the task for~e, the 
next step 1s to write precise specifications for the core Information base 
and the site-specific issues, creating a sort of map that lays out exactly 
what Information 1s to be collected and how. Specifications cover the 
content, type of measure, Item format, and respondent or source of 
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information. See Figure 1 for an example. 



Figure 1. 
Sample Specification 



CONTENT 



TYPE 



FORMAT 



RESPONDENT/SOURCE 



Overall perception 



questionnaire 



open-ended 



252 random sample 



of program s strength: 



of all students 



a. ways program has 
benefitted w ou 

b. doicribe one part 
that works well 

These specifications will guide development of the actual measures to be 
selected or created. Both activities, specifying information and creating 
or selecting measures, have been included in Step 4 because it is often 
easier to think of them simultaneously. Many people prefer to think in 
terms of sample items when considering each of the components to be 
specified rather than trying to consider them in the abstract. In 
addition, the process of creating measures often suggests new options for 
the components that were not previously considered, thus causing 
modification of the original specifications. 

When specifying the information base, it is important to note the 
variety of measures that may be used, such as questionnaires (open-ended as 
well as multiple-choice), interviews, observations, inspection of written 
records, and performance tests. There is also variety in the sources of 
data, all those who participate in the program, the most important of whom 
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are students, faculty, staff, and administrators. Occasionally there are 
others effected by the program, such as teachers who work with students 
after they have passed through the program, from whom the task force may 
want to collect informat. .t 1s usually informative to use 
triengulatlon of data— asking for the same Information from several 
different sources. Some types of measures, such as follow-up interviews 
witn students, may best be used by trained data collectors during a visit 
to the program site. Others may be used by regular program participants 
during the course of the program activities. The task force will probably 
want to make full use of all these options. 

Although many eva" tlons in the past have tended to focus on 
quantitative data and have eschewed using much qualitative data due to 
logistical problems, we would emphasize the value of both quantitative and 
qualitative data. However, it 1s easier to collect and use qualitative data 
1f it can be partly quantified on objective, standard forms. For example, 
demographics on the participants, a description of program services, and a 
comparison between the proposed program activities and actual services 
provided may be gleaned from records, interviews and observations. The 
information may then be used to fill out standardized, highly structured 
forms common to all sites. An example of one such form 1s appended to this 
report. 

Qualitative data on how Individuals experience the program may be 
collected via an open-ended questionnaire intended to allow Issues to 
emerge from comments made. (A sample questionnaire is appended to this 
report.) This data may then be categorized and talHeo on standard forms, 
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thereby quantifying the major concerns expressed. If the source of the 
information is known, interviewers may follow-up on some of the issues with 
a sample of the people who made relevant comments. 

Student outcome data may often be easily quantified, either through the 
use of existing tests (e.g. curriculum-based tests) or measures developed 
by the evaluator and task force specifically for this evaluation. 

In many cases the desire to gather more information than is necessary 
or feasible given constraints of personnel, money and time will remain a 
problem at this stage too. The task force, therefore, may again have to 
prioritize the list of proposed test items, Interview questions, and so 
forth, to obtain a reasonable amount to measure the agreed upon Information 
base. 

To save time and utilize available expertise, the evaluator can take 
primary responsibility for developing a tentative plan for the 
specifications, based on the task force's consensus on needs and priorities 
set in Step 3, with specific input from both policymakers and program 
specialists on the task force. The initial plan can be presented to the 
task force as a whole for review. Then the task force can select and 
refine those ideas which are most useful and generate additional Ideas for 
measures to ..nswer any remaining questions. This model, in which much of 
the generation of specifications (and measures) 1s done by the evaluator 
rather than by the committee as a whole simplifies the job of the task 
force and allows the evaluator to build 1n from the beginning certain 
necessary characteristics of a good evaluation (e.g. sound technical 
aspects of the measures and a realistic amount of information to be 
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gathered). At the same time, allowing task force members to refine and add 
measures gives users a sense of ownership of the measures rather than a 
sense of imposition by the evaluator. 

The final task in Step 4 is to select or create appropriate measures 
(test items, interview questions, observation protocols, questionnaire 
items, standardized record forms, and so forth) following the 
specifications just developed. Again, the major responsibility for 
development of the measures probably best lies with the evaluation 
specialist, consulting the task force for specialized Input, review and 
modification. The more precise the specifications drawn up earlier, 
the more straightforward is the development of measures. 

Step 5. Develop Procedures for Data Collection & Report of Results 
The next set of responsibilities for the task force is to oversee the 
development of a standardized system for using the measures developed in 
Step 4 to collect information and write reports for the individual program 
sites and for the policymakers. A system that is standardized across sites 
is most likely to provide the uniform information base desired for this 
type of evaluation. 

The specific subtasks include: 

a. specifying who will collect which data and when, 

b. training data collectors, 

c. orienting program participants to their role in the evaluation, 

d. planning the reports. 

A. Specifying who collects which data and when? 

How the data will be collected is largely a function of what the data 
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will be. If the task force wants to compare program plans with actual 
implementation and also obtain an in-depth, qualitative picture of 
participants' views of the program, it will probably be necessary to visit 
each program site to actually see the program 1n action and talk directly 
to participants. "One-shot" data, such as performance tests, can also be 
collected during this visit if desired. On the other hand, if only 
one-shot, paper and pencil measures are to be used, a site visit may be 
unnecessary, and the data may be able to be collected by the local program 
director at each site. If the task force needs data from two or more 
points 1n time, such as pre- and post-instruction measures, a multi -phase 
data collection process will be necessary. 

In most cases, a multi-phase process is most appropriate. First, a 
portion of the data 1s collected under the direction of the local program 
director at the Individual site, including pre- and post-instruction 
measures of student accomplishment, faculty, staff and student Impressions 
of the program via written questionnaires provided by the task force, and 
demographic characteristics of the students, which is available through 
school records. 

Second, a short (1- to 3-day), well-structured site visit by a small 
team of trained people examines the actual Implementation of the program 
compared to Its planned Intentions The team d^so personally Interviews a 
sample of the program participants. This approach allows them to follow-up 
on the Information obtained during the first phase of data collection, 
which the team has examined prior to their visit. During the site visit, 
team members meet regularly to share Impressions and cross-validate their 
observations. 

per 
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The use of a site visit team has a couple of advantages. It provides 
the opportunity to bring together data on the same issues from several 
different perspectives — a means of data triangulation. In addition, the 
structured, collaborative approach allows qualitative data to be 
systematically collected and organized for presentation within a very short 
period. 

Who comprises the team? Effective site visit teams utilized by Alkin & 
Stecher (1981) and Alkin & Ruskus (1985) consisted of three members: the 
team leader is a program analyst from the policymaking office; the second 
member is a program administrator who works at a different site from the 
one being evaluated; and the third member is either an external consultant 
or program staff member who has had experience or training 1n naturalistic 
observation. This seems to be an appropriate model, although the task 
force may wish to expand the team or make modifications in the members 
depending on the size and nature of the programs to be visited. 
B. Training data collectors. 

Regardless of who collects the data, it is crucial that they be given 
very specific directions. This is particularly true when a group of people 
are expected to work together as a team. A series of explicit handbooks 
drawn up by the task force for the site visit team members and the local 
program administrator can help assure consistency of data across sites. 
Such handbooks introduce team members to their purposes and process, 

> 

describe their specific duties, provide tentative schedules and standard 
data forms, and provide sample eports to illustrate how data may be 
analyzed and synthesized into the two reports. Some sample handbook 
components are appended to this paper. 
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C. Orienting program participants. 

In addition to training provided to the data collectors, there is also 
a need for orienting the program scaff at each of tne sites to be 
evu jated. They need to know quitj a bit of information in oroer to 
cooperate dr.d fulfill their roles: the purpose of the evaluation, how to 
administer any measures they may be responsible for, how large a sample to 
select and how to select it if measures are not being given to everyone, 
where and when to send the Information collected, expect to hear the 
results. If there is to be a site visit, the program staff need to learn 
what to expect of the visit and should be given relevant details about 
scheduling and logistics. They may also be given an opportunity to request 
that the site visit team investigate certain concerns they may have. 

D. Planning the reports. 

The task force must also plan how the data will be organized into 
reports for the two major purposes of this evaluation: policymakers and 
individual program sites. These reports will probably be somewhat different 
due to the different needs of each of these users. Individual sites need 
specific information for program improvement, while policymakers may need 
only a general picture of the overall orogram, with less or no detail about 
individual sites. The consensus on users 1 information needs, reached in 
Step 3, should guide the content of the reports. The following outlines are 
to be suggestive rather than prescriptive. These lists of suggested 
content should be used by the task force to generate a sample site repoi 
since in our experience, it 1s difficult to obtain a number of comparable 
reports when only an outline is provided to the authors. (See for an 
example Burry, 1985.) 
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The reports to each of the program sites may follow a common outline, 
such as the following: 

1. An executive summary Including the recommendations for specific 
program site 

2. A description of the evaluation purposes and procedures 

3. Qualitative report, Including: 

the setting, participants, qualitative methods used, positive 
findings, and areas for Improvement 

4. Report on program activities, Including data on demographics of 
students, staff allocations, and planned versus Implemented objectives 
(description, acceptability rating, and comments) 

5. Appendices with all measures used 

The report to the policymakers may Include the following types of 
Information: 

1. Executive overview of contents 

2. Description of evaluation purposes and procedures; demographic 
characteristics of participating program sites 

3. Decr1pt1on of what the program does: clientele, services, 
personnel 

4. Program strengths (overall); optionally, the most outstanding 
component of each program evaluated. 

5. Areas for program improvement: major themes of findings across 
sites; optionally, highlight specific program recommendations. 

6. Appendices with all measures used 
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Who is responsible for writing each of these reports? The site 
reports may be written by the site visit team at the end of their visit. 
The team members may jointly contribute their insights to a unified 
description of findings and recommendations for improvement. Responsibility 
for preparation of the report (including answers to both core and optional 
questions) probably best rests with the team leader. The team may then 
personally share the results with local program staff and school 
administrators to use for program improvement. The qualitative portion of 
the report may, however, be most easily written in the week following the 
visit since time will be short during the visit itself. If no site visits 
are conducted, the task force will have to select members to be 
responsible fcr receiving, synthesizing and compiling each site's report. 
In either case, the task force may want to provide each site report author 
with the description of the evaluation purposes and general procedures, 
since these should be common to all sites. 

The pc 14 :ymaker report is compiled by the director of the task force 
by aggregating the data from each of tte individual site reports. Details 
on individual sites may be included or omitted depending on decisions made 
earlier by the task force. Obviously, this report is done after data from 
all sites has been collected. 

One final note about the set of tasks in Step 5 is critical. Many an 
evaluator has felt that Murphy's Law must have been created to describe 
evaluations. The next step, piloting and refining measures and procedures, 
will be made much easier if ample time is allowed at this stage to 
anticipate problems and find solutions. 
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Step 6, Pilot and Refine Measures and Procedures 
Prior to actual implementation of the evaluation plan it is important 
to try out the measures and procedures. Good intentions cannot guarantee 
effectiveness. It is highly recommended, therefore that procedures be 
piloted at a site not scheduled to be evaluated at this time so that the 
results can be used solely to improve the measures and procedures. The 
more similar the pilot test is to the actual circumstances of intended use, 
the more likely th.se efforts are to reveal problems that may arise. 
Depending on the scope of the evaluation and the extent to which the 
measures and procedures are untried, it may be prudent to do a two-phase 
pilot. During Jie first phase, the emphasis may be on the measures and 
data collection procedures. In the second phase, revised materials may be 
tried out and reports written to determine if the entire process were 
capable of generating user-oriented reports. 

There are several problems which may arise and which the task force 
should be prepared for when piloting measures and procedures. Some 
participants in the evaluation do not do what they are supposed to, either 
because they fail to take the evaluation seriously or because they ^lid not 
understand ahead Df time how much time or effort it would take. Data 
collectors may misunderstand what they are required to do and may 
contaminate the results. Materials may arrive too late to be of use (e.g. 
pretests arriving r.idway through the instructional sequence). The measures 
may be targeted to the wrong people; they may include confusing and 
redundant items; and most commonly, the measures will be simply too long to 
effectively administer given the usual constraints. 
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In addition to piloting the evaluation procedures, 1t is also useful 
to try out the analyses and reporting strategies. These components are 
just as critical as the others to the overall success of the project. It 
1s entirely possible that a measure has been created that looks good, works 
well with the people taking it, and yet provides data that 1s unclear or 
difficult to Interpret. Such a problem may not be recognized until someone 
1s forced to synthesize the findings 1n a report. In addition, this step 
will Indicate whether adequate reports are likely to be produced with the 
planned materials, Information, and procedures. 

Step 7. Collect Information 

Once the procedures and materials have been piloted and refined, the 
real e valuation can begin. At this point the task force's duty 1s to 
oversee the implementation of the plans and help to solve additional 
problems as they may arise. 

Evaluations in general and site visits in particular sometimes evoke 
fear of "final certification" or loss of funding. Results will be more 
useful if the task force is aware of this possibility and can reassure 
sites abDut the r*al purposes of the evaluation. They can emphasize tha 
the visit is a chance to supplement other data with interviews and direct 
observation of the program and that 1t also provides an opportunity for the 
program staff to request a closer look at certain elements of the program. 
Finally, the visit allows for dialogue about findings rather than one-way 
communication. 

Step 8. Prepare Reports 
The final duty of the task force is to oversee preparation of s 4 te 
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reports and to prepare the final report to the policymakers. If the task 
force has done a good job of planning the contents of and procedures for 
writing these reports 1n Step 5 and has piloted and modified the report 
writing plans 1n Step 6, this final step should proceed without difficulty, 

SUMMARY OF CRITICAL CONCERNS 

1. The task force must agree on a uniform Information base to meet the 
designated needs. In one evaluation new members were added to the task 
force midway through the process, and these new members questioned 
decisions that had been made without them. This caused the group to stall 
a number of times to educate these members or to rehash old ground. In 
addition, "agreement" seemed to have been reached, only to be rescinded 
later by some members who changed their minds or who had never 
wholeheartedly agreed 1n the first place but had not aired their concerns 
earlier. Utilize oeer pressure by members who are "on track" to persuade 
others to contribute constructively. 

2. The task force must Involve the key users of the evaluation 
results, who should understand that the results are truly Intended to be 
used, not just filed away somewhere. This requires a mind set among local 
program staff to use data rather than mere Intuition for program 
Improvement. Ultimately, the usefulness of data for Improving local 
programs rests with the local users— their Inclinations, attitudes and 
schemas. Are they truly trying to Improve the program, or 1s personal 
survival their first priority? 

3. The measures must be relevant to the agreed upon focus, technically 
adequate, able to produce meaningful and useful results, and be acceptable 
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to the users. This will require a delicate balance of Input from users and 
professional evaluators. In a previous evaluation where measures were to 
be developed primarily by the task force members with guidance from the 
evaluation expert, the resulting measures were wildly disparate 1n size, 
scope and technical properties. Yet the evaluator felt constrained by 
needing task force members 1 ownership of the measures. Perhaps such 
ownership can be obtained through review rather than painstaking 
development of specifications. 

4. Standardization of procedures and careful attention to logistical 
details will help assure comparable, meaningful Information with fewer 
negative side effects for evaluation participants. 

5. Realistic expectations regarding what can be accomplished given the 
available time, resources and personnel are critical to success. Task 
force members may be reminded of the Importance of a few good measures, 
done well, with clear Implications for action over a large, messy 
hodgepodge that 1s ultimately forgotten. 



33 



27 



Conclusions 

In this paper we have provided a model for evaluating programs for two 
purposes simultaneously: for program Improvement and for policy making. 
This dual focus approach has two major advantages by providing a common 
Information base for decisions at both ends of the continuum, 1t helps 
ensure that these decisions are 1n harmony with each other. In addition, 
this approach conserves time and effort of staff and students alike by 
making one data collection effort serve two purposes. 

This user-oriented model reflects many years of evaluation experience 
and has been used successfully 1n several educational settings. It relies 
on a task for;e of evaluation users, who are open to the possibility of 
change and are committed tf using the results. As a group they must reach 
consensus n their purposes, desired Information base, and procedures, and 
then see that the plan 1s carried out. 

The model evaluation plan presented here consists of eight steps, as 
follows: 

1. Identify Evaluation Users and Their Needs 

2. Form Task Force 

3. Reach Consensus on Information Needs 

4. Specify Information Base and Develop Measures 

5. Develop Procedures for Data Collection and Report of Results 

6. Pilot and Refine Measures and Procedures 

7. Collect Information 

8. Prepare Reports 

While this plan was based on evaluation experiences 1n education, 1t 
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should also prove useful in other fields 1n which there 1s a need to make 
data-based decisions to Improve local programs as well as to create policy 
that serves a number of such programs. 
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Appendix A 

Sample Standard Form for Quantitative/Qualitative Data 


Part IV: Staff Allocations: Indicate only people for whoa there Is a discrepancy between proposed and actual activities 


Name 


FTE 


Discussion of Discrepancy (Major variances by component #) 


Rec i 
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Appendix B 



Confidential Student Questionnaire 



Directions : Please answer the following questions. To maintain 

confidentiality, please seal your completed questionnaire 1n 
the attached envelope. Sealed questionnaires will be read by 
review team and will not be shown to local students, staff, or 
administrators. Please be candid ~ your comments can help to 
Improve the project. 

Thank you. 

1. In what ways do you feel the project has been of greatest benefit to 
you? 



2. Describe one part of the project here that you feel works particularly 
well. 
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Student Questionnaire (Cont'd) 

Why do you think It's effective? 



3. Describe one part of the project here that 1s 1n need of Improvement. 



Why do you think 1t needs to be Improved (What's wrong with 1t)? 



Please specify your major area of study 



Please Indicate services you have received from the project: 

Financial aid Counseling Tutoring Recruitment 

Employment assistance Chlldcare Transportation 

Your name (Optional) 
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Appendix C 
CHECKLIST 
Team Leader 



Prior to tne Site Visit 

□ Identify college to be visited and select individuals to serve as 
Team Members B and C. 

1 "[schedule visit and make travel arrangements. 

Q] Prepare Site Visit Packet. 

□Mail Site Visit Packet and Handbook with supplements and proper 
site visit schedule to Team Members B and C. 

□ Mail to Program Director thj "Director's 6uide M and tiultiple cop.es 
of Confidential Student and Faculty/Staff Questionnaires, and 
survey of student coals. 

I I Check with Director to assure that student goals survey has been 
' — 1 conducted and confidential questionnaires distributed and 
returned. Remind Director to fill out Summary of Project 
Accompli shment* c orm and Student Population form which are part of 
the "Director 1 jide." 

□ Be certain that Director mails completed confidential 
questionnaires to Team Member C at least ten days prior to site 
visit. 
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During the Site Visit 

Coordinate all activities Including: 

□ Conduct brief team meeting before the visit to get oriented to 
procedures. 

j^j Informal Introduction. 

QjPlannlng Meeting (Team Members A, B, and L rrect with Director). 

QjAssign Interview Tasks (Tea* Members A and B). 

£2 Camous Orientation {Team Members A, B, and C tour campus). 



□ 



Gather and Record Data (Team Members A and B jo^.tl* interview 
Director; Team Members A, B, and C separately .nteidew other staff 
members and students, observe project activities, and review 
documents). 
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□Prepare for exit Interview (Team Members A, B, and C prepare 
recommendations and compose brief description of findings). 

□Conduct Exit Interview (Team Members A, B, and C with college 
president and administrators, Director and senior project staff. 



After the Site Visit 

| \ Prepare final report. 

□ Distribute final report to Director, college president, college 
administrator who supervises Director, to each of the other team 
members, and to Task Force Director (for synthesis in Report to 
Policymakers). 
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Time 

Evening Prior 
to Day 1 



Day 1 

8:^0-9:00 am 



9:00-10:00 am 



10:00-12:00 am 



12:00-1:00 pm 
1:00-2:30 pm 

2:30-5:00 pm 



Appendix D 
Tentative Schedule 
(Two-Day Site Visit) 

Activities 
- Arrive at hotel 



Team meeting to review 
site visit schedule and 
responsibilities 



- INFORMAL INTRODUCTION 

- Meet with project staff 

- Explain purpose of site 
visit 

- Coffee 

- PLANNING MEETING: 

- Meet with Program Director 
to review project data and 
to determine interview 
sources 

- ASSIGN TASKS 

- CAMPUS ORIENTATION: 

Tour of campus and E0PS 
facilities. Visit with 
President or his designee 
to comment on purpose of 
of visit. Scheduled meeting 
and short interview with 
immediate supervisor of 
Program Director 

- Lunch 

- GATHER AND RECORD DATA: 

Interview Director 
Interview other staff 
members and students 

- GATHER AND RECORD DATA: 

Interview other staff 
members and students 



Team 
Members 

A+B+C 



A+B+C 



A+B+C 



A+B+C 



A+B 
A+B+C 



A+B 

C 



A+B+C 

(separately) 
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Time 

8:00-9:30 pm 

Day 2 

8:30-11:00 am 

11:00-12:00 am 
12:00-1:00 pm 

1:00-2:30 pm 

2:30-3:30 pm 

3:30-4:30 pu 

4:30 pm 



Team 

Activities Members 

- Team meeting to review A+B+C 
progress, discuss prelim- 
inary recommendations, and 
coordinate activities for 
Day 2. 



GATHER AND RECORD DATA: 
Interview staff members 
and students, observe 
project activities, and 
examine documents 

GATHER AND RECORD DATA: 
Interview Director 



A+B+C 

(separately) 



A+B 



Luncheon meeting among team 
to monitor status of site 
visit and plan final activities 

GATHER AND RECORD DATA: 

Final Interviews, observations 

etc. 

Team meeting to prepare final 
recommendations and summary 
of findings 

EXIT INTERVIEW with college 
presdent and administrators, 
E0PS Director, and senior 
pr>jec* staff 



A+B+C 



A+B+C 

(separately) 



A+B+C 



A+B+C 



- Depart 
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INTRODUCTION 



Whenever one presents the results of a questionnaire survey, there 1s 
always someone who says "But those are only opinions". If the results come 
from a survey of students, the put-down response 1s "But those are only 
students ' opinions", as 1f, coming from students, the results are even less 
believable. If the comment comes from someone 1n the "hard" sciences, 1t 
1s likely to be "But you only have 'soft 1 data". 

It's Interesting that this sort of knee-jerk disbelief does not 
automatically occur 1n response to other surveys. The Census Bureau 
conducts many surveys that ask about people's opinions and plans. There 
are surveys to estimate consumer confidence which are taken seriously by 
economists and entrepreneurs. Political opinion surveys are carefully 
studied by candidates for office. Opinion surveys are an Important aspect 
of market research. There 1s, of course, a certain skepticism about the 
credibility of some self-reports to the Internal Revenue Service. But on 
the whole, opinion polls, survey research, and questionnaires are widely 
accepted methods of Inquiry, and certainly a very significant feature of 
scholarship 1n the social sciences. 

Opinion polls and attitude surveys, like other Inquiries, are subject 
to errors of measurement. For more than fifty years there has accumulated 
a very large body of research on possible sources of error, and on ways to 
estimate reliability and validity. The Public Opinion Quarterly regularly 
publishes scholarly articles on the methodology of polls and surveys. The 
major polling agencies are especially sensitive about the accuracy and 
validity of their reports. Some of the jest known survey centers are 
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university-based as the National Opinion Research Center at the 
University of Chicago, and the Institute for Social Research at the 
University of Michigan. 

In higher education, and In education generally, questionnaires are 
quite common. There has also accumulated over a period of years a body of 
research on the credibility of students answers to questionnaires. The 
present report on the credibility of student self-reports Is a preliminary 
document that should, and perhaps may, become a more thorough and scholarly 
document at some future date. Meanwhile, we aim to present a few 
highlights from the large literature on measuring attitudes and other 
subjective phenomena, note some of the accuracy checks that have been made 
with respect to college student questionnaire responses, and then examine 
briefly the features of two current questionnaires for entering college 
students and explore more extensively one current questionnaire for 
undergraduates to Illustrate a variety of reliability and validity 
estimates that can sometimes be produced to demonstrate the credibility of 
students answers. 
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PART 1 

ISSUES, ANSWERS, AND ADVICE 

The Russell Sage Foundation has recently published a definitive two 
volume document entitled Surveying Subjective Phenomena. (Turner and 
Martin, Editors) 1984. For anyone who wishes to review the literature of 
research on this topic, those two volumes are a fairly complete answer. In 
addition, the Russell Sage Foundation has also published a book by one of 
the most highly regarded scholars, Otis Dudley Duncan, Notes on Socia l 
Measurement: Historical and Critical. 1984, which deals with the whole 
domain of counting and classifying demographic and other elements, from 
antiquity to the present. 

In 1976 the College Entrance Examination Board published a monograph 
by Leonard Balrd, Using Self-Reports to Predict Student Performance , which 
reports much of the evidence from college student surveys about the 
accuracy of their responses to questionnaire Items, as well as their 
utility for prediction. 

Part 1 of this report Is not a review of the literature In the usual 
sense. No attempt Is made to cite chapter and verse from dozens of 
studies. Rather, everything (except as may be subsequently noted) that 
will be men tinned comes from one or more of the four major sources just 
cited. What follows, then, Is my summary of what I regard as a few 
highlights from the literature, plus some of my own contributions to that 
literature over the past 50 years. 
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Varieties of Self-Reports 



Some self-reports merely ask for obvious, easily verifiable 
Information, such as age, sex, marital status. It 1s a subjective or 
Individual answer to an objective question. At the other end of the 
spectrum are questions and answers both of which are entirely Interpreted 
by the Individual. A good example 1s the following question: "Taken all 
together, how would you say things are these days — would you say that you 
are very nappy, pretty happy, or not too happy?" An example from a survey 
of college alumni 1s the following: "What 1s your present feeling about 
your college? — strong attachment to 1t, pleasantly nostalgic but no 
strong feeling, more or less neutral, generally negative, thoroughly 
negative". The meaning of the question and of the response 1s determined 
by the respondent, and can be directly known only by the respondent. 

In one part of the appendix to Volume 1 of the Russell Sage report 
there 1s a "Scheme for classifying survey questions according to their 
subjective properties" (pages 407-431). The main categories of this scheme 
Illustrate the varieties of self-reports one encounters 1n surveying 
subjective phenomena. There are three dimensions. The first 1s the 
referent of the question: objective versus subjective events. Objective 
questions refer to events that can be externally observed. Subjective 
questions refer to Internal conditions, Intuitions, beliefs, etc., which 
are directly knowable only by the Individual. The second dimension 1s the 
nature of the judgment. Such judgments might Involve beliefs, 
attributions, or valuations, and they Involve different Intellectual 
tasks. Simple judgments about the occurrence of events primarily Involve 
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recall. Attributions require generalizations and Inference. One finds 
very generalized referents such as "most people", "all 1n all", "people 
running the country today", "most faculty members", etc. The 
Interpretation of answers 1s complex and surely suggests the Importance of 
skepticism. Valuations Include questions about preferences, likes and 
dislikes, approval ratings, attitudes toward people, groups, organizations, 
policies, subjective sentiments such as confidence ratings, satisfactions, 
problems and worries. The third dimension 1s the object of the report: 
self versus other. Is the respondent being asked to report about himself? 
If so, do people tend to present themselves 1n a good light? How do these 
self-perceptions Influence one's perception of others? 

These three broad categories, albeit overlapping 1n some respects, are 
useful to keep 1n mind as one examines the content of questionnaires: the 
referent of the question, the nature of the judgment, and the object of the 
report. 

Errors of Measurement 

In questionnaire surveys of college students the chief source of 
unrepresentative results are the nature and size of the sample, and the 
proportion of people who return the questionnaire. Students 1n a large 
Introductory psychology course are often asked or required to respond to 
some questionnaire. They, of course, are not a representative sample of 
anything. For relatively small colleges, the best advice 1s to give the 
questionnaire to everyone, thus bypassing the sampling problem. In big 
universities, the task of having all entering freshmen respond to a 
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questionnaire 1s never successfully completed. If one can get two-thirds 
or three-fourths of the population one 1s doing rather well. There are 
good studies that have obtained data from a broad assortment of students 
and institutions; but nothing comparable to a national public opinion poll 
1n Us representativeness. The more significant problem, however. 1s 1n 
the response rate. Whether questionnaires are distributed via the U.S. 
Postal Sevlce, or whether they are put 1n a campus mailbox, many are never 
returned. 

In a national questionnaire survey of students and alumni which I 
carried out 1n 1969. Involving random samples at about 75 colleges and 
universities, the median response rate to the freshman questionnaire was 
801. for the upperclassmen questionnaires the median response rate was 66X. 
and for the alumni samples the median response rate was 581. The 
questionnaires, each about 16 to 20 pages 1n length, were attractively 
designed and printed; most colleges used one followup reminder; and for the 
alumni samples there were two followup reminders. 

Even 1f one had returns from everyone the basic conclusions would not 
change significantly; but probably in all questionnaire surveys there 1s 
some selectivity or bias among those who respond. In the 1969 study the 
poorest rates of return from freshmen and upperclassmen came from the large 
Institutions; but 1n the alumni questionnaire the differences 1n return 
rates were not related to size, they were related to Institutional 
selectivity and prestige. In the elite categories, only 2 1n 20 (10*) had 
an alumni response rate of less than 50X; 1n the middle category 
scholastlcally. there were 10 of 39 (26X) with a response rate of less than 
50X; and 1n the least selective category, there were 5 of 15 (33X) with 
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fewer than 50X returns from their alumni. 

In two recent questionnaire surveys of UCLA undergraduates., the 
response rates have been between 451 and 50X There are. of course, ways to 
Increase the rate of return of mailed questionnaires. Unfortunately, for 
academic researchers, they are very costly and the money 1s not 
forthcoming. 

Unlike the usual procedure 1n academic surveys, the national opinion 
polling agencies collect their data by Interviews. The carefully designed 
stratified area sampling techniques do. 1n fact, produce reasonably 
reliable and valid results. The magnitude of non-response 1s minimal 
because the Interviewers' s Job Is to get everyone who fits the sample 
specifications. 

On several past occasions I have suggested that periodic polls of 
college students might be very worthwhile. But they would require 
developing an adequate base for sampling, and this does not now exist. The 
carefully designed sampling procedures, and the resulting national samples 
for public opinion polls, are not applicable to the college population. 

There are several other aspects to the present topic of measurement 
error. These relate to the estimation of reliability. Does one get 
similar answers to the same questions from comparable samples? In a 
test-retest situation, do people give the same answer the second time that 
they gave the first time? Do slightly different questions about the same 
topic result 1n generally similar responses? Most surveys 1n social 
science and 1n higher education do not report answers to any of these 
questions, and presumably do not collect evidence about any of these 
matters. But they should. And at least periodically they have. 
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In 1948 a 16-page questionnaire was mailed to a sample of Syracuse 
University alumni. The questionnaire Included two types of Items which 
were subsequently readmlnlstered to a small sample. The questionnaire 
contained eleven Activity Scales of eleven Items each, labeled Politics, 
C1v1c Affairs, Religion, Art, Music, Literature, and Science. The subjects 
checked each activity they had engaged 1n during the past year. The scales 
were Guttman-type scales 1n that participation 1n the more dlflcult 
activities tended to subsume participation 1n the easier anC more common 
activities. The score on each scale was simply the number of activities 
checked. Then there were nine Opinion Scales of six Items each, labeled 
Politics, C1v1c Relations, Government, the World, Philosophy, Art, Music, 
Literature, and Science. The statements 1n the opinion scales were written 
to reflect basic concepts or generalizations about the topics, 
generalizations reflecting a consensus of experts 1n the field, so that 1t 
was possible to score each scale simply by counting the number of > 
statements on which one's opinion agreed with the opinions of the experts. 
Each statement was answered on a five point scale, from Strongly Agree to 
Strongly Disagree. Six months after the initial sample of 2500 had filled 
out the questionnaire, a second copy was sent to a small group of 120, 
receiving 68 1n return. The test-retest consistency of scores over this 
six-month Interval was computed. For the Activity Scales, the correlations 
ranged Uom .70 to .89, with a median of .83. For the nine Opinion scales 
the median test-retest correlation was .65, with seven falling between .60 
and .70, and two much lower ones of .40 and .31. Consistency of responses 
was also checked Item by Item. For the Activity Items, the averge percent 
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of Identical responses was 85, with a range from 83 to 87. For the Opinion 
Items the average percent of Identical responses was 75. with a range from 
68 to 84. The above test-retest data were reported In an article by Pace, 
"Opinion and Action: A Study In Validity of Attitude Measurement", 
Educational and Psycholo gical Measurement. Vol. 10. No. 3, 1950. pages 
4' 1-429. 

Th« ACT Evaluation/Survey Sevlce, Users Guide, 1981, reported 
te.t-rateit results on ACT's Student Opinion Survey for a group of students 
at one university who responded to the questionnaire a second time 
approximately two weeks after the Initial r ;ponse. The average percent of 
Identical responses on the two admlnl stations was 98Z for demographic 
background Items (age. race, sex. etc.). 90X for other background Items 
such as hours worked per week, occupational plans. et-. f and 93X for Items 
about the usage of college programs and services. For "Satisfaction" Items 
(responses on a five-point scale from Very Satisfied to Very Dissatisfied' 
referring to such matters as academic aspects of the college environment, 
rules and regulations, facilities, college services, etc.. the percent of 
Identical item responses was typically about 64X. and the percent of 
re«oonses within one scale point of the Identical response typically about 
95Z. 

In the American Council on Education Pesearch Report, Vol. 7, No. 2, 
1972 by Boruch and Creager, entitled Measurement Error In Social and 
Educational Survey Research, ty 0 examples of test-retest comparisons are 
cited. Cie example administered a questionnaire twice, with six weeks 
Intervening, to a group of 107 college students. Questions about students 
previous achievements resulted In 90X to :00X agreement. Answers to other 
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facts — such as fatr-r' s edMcatlon and occupation, high school grades, 
etc., had agreement percentages from 741 to 92X. Att1tud1nal Items, and 
Questions about future plans typically Involved agreement 1n the 5O-70X 
range. The other example was the readmlnl strati on of ACE freshman 
survey questionnaire to 202 students following an Interval of two to three 
weeks. Test-retest correlations for different types of Items were as 
follows: demographic characteristics, mostly .96 to .99; sources of 
financial support, mostly .85 to .88; self-reported a«.,r1butes of parents, 
mostly .60 to .82; Items estimating the chances of future events (such as 
graduating with honors, joining a fraternity or sorority, falling one or 
more courses, chang.ng career tholce, etc.), mostly .58 to .88 with a 
median of .78; Items about life goals such as the Importance of being very 
well-off financially, raising a family, keeping up with political affairs, 
helping others 1p difficulty, mostly from .65 to .87 with a median of .73; 
attitudes toward the Importance of various federal actions such as 
pollution control, school desegregation, veterans benefits, consumer 
protection, correlations ranging from .41 to .83 with a median of .63; and 
Items about attitudes toward various campus and social Issues such as 
faculty promotions should be based on student evaluations, marijuana should 
be legalized, with test-retest correlations ranging from .57 to ,8C with a 
median of .66. 

Both the ACT and ACE reports sh that the greatest variability 1n 
responses are found 1n relation to questions that are ambiguous, or about 
topics which students may not have given much prior thought or concern , 
about, attitudes which are themselves subject to various Interpretations. 
In some cases, the test-retest correlations are ^ow enough to raise doubts 
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about the value of the responses, especially when the test-retest Interval 
1s only 2 to 6 weeks. For the more specific Items, consistency of 
responses was quite high. 

In public opinion surveys there have been some examples of comparing 
the results to the same questions when asked by different survey 
organizations. The closest or most carefully controlled conditions are 
called tandem surveys. In one such tandem survey, N0°C and Roper each drew 
probability samples and proceded to administer the survey 1n their 
customary fashion. This was a survey about public use of and attitudes 
towards television. Differences In the results were small; but there was a 
clear effect related to how the organization determined the "don't know" 
responses. On 52 comparisons, NORC had fewer DKs on 42 Hems, Roper fewer 
on 4 Hems, with no differences on the other Items. In another study, a 
survey about public attitudes and knowledge concerning survey practices, 
the sample was drawn by the Survey Research Center, and the cases randomly 
aligned to SRC and Census Bureau Interviewers. In general, the results 
were fairly similar. However the Interviewee refusal rate was 61 to the 
Census Bureau Interviewers and 13Z to the SRC Interviewers. 

A summary tabtlatlon reported 1n Volume 1 of the Russell Sage 
publication, of 126 Instances 1n which the same questions were asked by 
different survey measurement programs at about the same time shows that 1n 
45 of the Instances there were differences beyond the level typically 
allowed for scmpHng error. Such differences could have come from many 
sources — context, Interviewer effects, training and staff differences, 
etc. Some of the differences were clearly attributable to how DKs were 
handled. Variations 1n practices produce differences 1n the products; but 
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mo«t of these differences are relatively small. When the conditions are 
most comparable, as 1n tandem surveys, the results are highly congruent. 

Errors of Substance 

Whether peop> report accurately about their conditions or their 
behavior 1s, In one sense, an error of measurement and 1n another sense an 
error of substance. In surveys of college students there 1s a good deal of 
evidence that self-reports about their school grades, and about prior 
eccompHshments are very accurate. Much of this literature has been 
summarized by Leonard Balrd 1n the monograph he wrote for the College Board 
In 1976. Are student's self-reports of their grades accurate? Balrd 
himself found that the correlation between college-reported and 
student-reported grades was genrally about .87. In a study of 
self-reported and transcript-reported grades, by Nichols and Holland In 
1963 among National Merit Scholars, cited by Balrd, the correlation was 
.96. Maxey and Ormsby 1n 1971 reported correlations between self-reported 
and school -reported grades 1n a sample of nearly 6000 students 1n 134 
schools to be or. the average 1n the mid eighties. They found that 981 of 
the students' reported grades were accurate within one grade. Balrd 
concludes from many studies that "research accumulated over 30 years, using 
various methods, 1n samples of grade school students, high school students, 
college applicants, junior college students, four-year college students, 
and professional school students, adds up to one conclusion: students' 
reports of their grades are about as useable as school -reported grades", 
(page 8). Moreover, self-reported grades predict future grades as well as 
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or better than college entrance tests of academic ability. It seems fair 
to conclude that, at least for some kinds of questions, errors of substance 
1n the answers are minimal. 

The data from the above studies are a good example of what one can 
expect when the questions are clear and specific, and when the response 
options are equally clear and specific. Students know the definition of 
grades and they know their own grades. Consequently, one can have 
confidence that the subjects can answer the questions. But In many surveys 
no such clarity 1s evident. 

Evidence from the larg r survey research literature also confirms the 
accuracy of self-reports about various specific conditions or behavior. 
For example, correlations between employers records about wages, duties, 
etc., and application blanfc work histories were generally .90 or greater. 
Adults reports of whether they owned their home were 96Z accurate, had a 
valid library card 87Z accurate. One needs to be reminded here, that 
"official records" are not always 100X accurate. 

Perhaps one of the most serious errors of substance arises from 
variations 1n the content, or wording, <,r the questions, and from the 
context 1n which the questions are l.jC There are some classic examples 
of this. The following question was asked 1n a national sample poll: "Do 
you think the United States should let Communist newspaper reporters from 
other countries come 1n here and send back to their papers the news as they 
see 1t?" Half the questionnaires asked this question after another 
question on whether the Soviet Union should allow 1n American newspaper 
reporters; and the other half of the questionnaires asked the questions 1n 
the reverse order. When the question about communist reporters was asked 
first, 551 of the people agreed; but when the question about American 
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reporters was asked first, 75X agreed. Or, consider the following two 
questions: 1) Do you approve or disapprove of a married woman earning 
money In business or Industry If she has a husband capable of supporting 
her? (65X of a national sample approved); Z) If there 1s a limited number 
of jobs, do you approve or disapprove of a married woman earning money 1n 
business or Industry when her husband 1s able to support her? (Only 361 
approved!) Here 1s another example of different answers from slight 
differences In wording. "Do you think the United States should forbid 
public speeches against democrary?" (Yes, 54X.) Do you think the United 
States should allow public sp.eches against democracy?" (No, 751). 

Another type of error, potentially causing substantive or Interpretive 
dlfflcltles, 1s the use of response options that each person Interprets 1n 
his own way. Examples of such response options are the use of words or 
phrases such as frequently, occasionally, rarely, most of the time, very 
much, quite a bit, usually, seldom, a great deal, very little, etc.. 
Presumably words such as always and never mean the same to everyone. But 
how often 1s "often"? And how much 1s "very much"? 

Pace and FMedlander, "The meaning of response categories: how often 
1s occasionally, often , and very often?", Research 1n Higher Education . 
Vol. 17, No. 3, 1983, addressed this issue using data from the College 
Student Experiences questionnaire. Participation 1n various college 
activities were Initially Indicated by the responses "never", 
"occasionally", "often" or "very often". Later 1n the questionnaire seven 
of the same activities were responded to as follows: For each of the Items 
below, fill in one of the spaces to the left which best Indicates the 
number of times you have engaged 1n the activity. These more specific 
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responses were: "never", "once or twice during the year", "about three to 
six times during the year", "about once or twice a month", "about once a 
week" and "more than once a week". By this means we were able to show what 
students meant (number of times) by the more general words. The results, 
9 i one would expect, revealed considerable overlap by what was meant by 
occasionally, often, and very often. But there was also a clear 
concentration or clustering of responses as one moved from occasional to 
often, and from often to very often. The meaning or definition of these 
general descriptors was different, depending upon the topic; but within the 
same topic the differences between colleges or types of students were quite 
small. In general, the definition of "occasionally" at one college was 
similar co Its definition at other colleges, given the same topic. 

Every respondent knows perfectly well that "very often" 1s more than 
"often", and that "often" 1s more than "occasionally". Thus, the 
direction of the scale 1s recognized by everyone. But the specific meaning 
attached to the labels 1s an Individual judgment. There were few obviously 
Implausible responses — such as students who Initially said "occasionally" 
or "often" but later said "never"; or students who Initially said 
"occasionally" but later said "more than once a week". These discrepancies 
constituted from Z% to 10X of the total responses. 

Comparative judgments of this sort necessarily reflect some reference 
group 1n the mind of the judges. On this questionnaire, we assume that the 
college peer group 1s the reference group, and that the answers reflect an 
awareness of what 1s customary In one's own behavior and 1n the behavior of 
the peer group. 
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The point of these observations about the subjective definition of 
response choices Is that one should get, If at all possible, some sort of 
evidence about what people mean by their choices. This same advice apples 
to opinion polls which ask about degrees of happiness, satisfactions, 
confidence, or other subjectively defined responses. 
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PART 2 

THREE COLLEGE QUESTIONNAIRES 

Efforts to evaluate the Influences of college on students' learning 
and" development should draw upon many sources of evidence. For .much of 
this relevant evidence the students themselves are the source; and the most 
common method for obtaining that evidence 1s a questionnaire. 

Here, for example, are four crucial questions. 

1. Who goes? What do we know about the entering students: their 
high school record and test scores, their family background, financial 
status, their Interests, expectations, aspirations, past achievements, 
etc.? Some of this Information can be obtained from records, but some wn 
be obtained only by asking the students themselves. 

2. What do they do after they get there? Some answers can be 
obtained from college records — such as, campus residence and major field, 
but for other sorts of behavior — such as the time and effort devoted to 
study, contacts with faculty, Involvement 1n extra-curricular activities, 
use of the library, etc. — the answers can only come from students' 
responses to questionnaires. 

3. What's 1t like? Physical facts — such as big school, small 
school, and big dty, small town — are Important. So also are students' 
perceptions of the campus environment or atmosphere. Wnat 1s stressed? 
What 1s expected? How do people relate to one another — friendly, 
supportive, cr not? Answers to these questions can only come from the 
students themselves. 

4. What do they get out of 1t? Knowledge, basic skills, and 
abilities relevant to a career, relevant to personal maturity and life 
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satisfaction, relevant to civic enlightenment — these are some of the 
possible and intended results. Achievement tests, aMlty tests, 
personality tests, etc. can provide some of the answers. It may also be 
Important to find out what the students themselves think they got out of 
college; and here again one relies on responses to questionnaires. 

Questionnaires can, and I think should, be regarded as a form of test 
or measuring Instrument. Many questionnaires are 1n fact regarded as tests 
by those who construct them. So, *ee have tests of attitudes and beliefs, 
vocational Interests, pesonallty traits, etc.. A variable or dimension to 
b measured 1s defined, sets of Hems are developed to measure It, and the 
reliability and validity of the results are determined. The process 1s 
similar to the construction of an objective achievement test, or a test of 
developed abilities such as the Scholastic Aptitude Test. Attitudes, 
Interests, beliefs, etc., are subjective phenomena. The answers one gives 
to a question about Interests or opinions 1s determined by the Individual. 
The student decides whether he agrees or disagrees with some statement, or 
likes or dislikes some activity, or person, or condition. The good 
published tests of personality, Interests, or values provide extensive data 
regarding their reliability and validity — tests such as the Minnesota 
Multiphasic, the Omnibus Personality Inventory, Holland's Vocational 
Preferences Inventory, the Allport-Vernon-Llndzey Study of Values. In some 
tests of this sort, the authors have Included a few Items to detect whether 
a student 1s giving false or Improbable answers — a practice which 
recognizes the Importance of estimating the credibility of self-reports. 

Many of the questionnaires used 1n studies of higher education are not 
designed as tests 1n the classical sense. They consist of sets of Items, 
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often grouped or classified under certain topics, but having no underlying 
or scorable dimension. One finds for example, various Hems about students 
use of counseling services, or various Items about students opinions of 
teaching practices, or various Items about students attitudes regarding 
political and economic Issues. The Items are no doubt regarded as 
Interesting and the answers useful to know. But the content 1s best 
described as a classified catalogue rather than as a theoretically or 
conceptually based set of dimensions or characteristics. The value of the 
question and the credibility of the answer has to be examined Item by 
Item. There 1s nothing Inherently unreliable or Invalid about a one-Item 
test. Most public opinion polls are really one-Item tests. But 1t 1s 
Important to realize that variations 1n responses are often caused by 
variations 1n the phrasing of the question. Slight changes 1n wording can 
produce significant changes 1n responses. Consequently, the meaning of the 
answers rests on a slender base. 

To begin Part 2 we briefly report a tabulation of "missing cases" 1n 
three questionnaires for college students. The results Illustrate some of 
the principles and advice given 1n Part 1, and serve to confirm, with these 
three current cases, the merit of that advice. Then, the ~»1n content of 
Part 2 1s a detailed examination of one questionnaire to Illustrate some of 
the Internal and external checks that can be made to tssoss the reliability 
and validity of students responses. The content of this one Instrument — 
Pace's College Student Experiences Questionnaire — makes meaningful cross 
checks possible, for 1t bears upon all four of the topics noted 1n the 
Introduction to Part 2: Who goes? What do they do after they get there? 
What's it like? and What do they get out of 1t? 
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Missing Cases: What types of questions are not answered? 

To provide some current Illustrations of non-response to questionnaire 
Items we have examined two widely used Instruments, each having the same 
general purpose and each Intended for the same type of population. The 
first 1s the Entering Student Survey, distributed by the American College 
Testing Program. The second 1s the Student Information Form, distributed 
by the UCLA Higher Education Research Institute. 

Both of these questionnaires are Introduced with assurances regarding 
the confidentiality of the students 1 responses. The HERI questionnaire 
says "Identifying Information has been requested 1n order to make 
subsequent mall followup studies possible. Your response will be held 1n 
the strictest professional confidence". The ACT questionnaire says, "The 
Information you supply on this questionnaire will be kept completely 
confidential. Your name, address, and Social Security number will enable 
college officials to Identify your responses and to contact you directly. 
The data you supply will be used for research purposes and will not be 
Individually listed on any report. If, however, any question requests 
Information you do not wish to provide, feel free to omit 1t." 

Both questionnaires have many similar and 1n some cases Identical 
Items, for example: age, race, sex, marital status, planned college 
residence, high school grades, planned college major, planned occupational 
choice, sources of funding, reasons for going to college. Straightforward 
Identification questions, and questions about specific activities, reasons 
for going to college, etc. are typically omitted by fewer than 41 of the 
cases, and often by fewer than 2f. The questions which are omitted by the 
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largest percentages of respondents are ones related to money, religion, 
expected major field and occupation, and assorted Hems about personal and 
social values. 

On the HERI questionnaire there are typically about 12% to 13X who do 
not answer the Items about parents Income, and sources of funding for 
college. Many of those Items -identify specif 1c. 4ollar amounts ~ parents 
total Income — or a specific fact ~ listed as a dependent on federal 
Income tax return. No doubt 1n some Instances the students do not know the 
answers; &nd perhaps 1n other Instances they regard the question as 
Inappropriate. The ACT does not ask about dolors; 1t asks whether various 
sources of funding are a major source, minor source, or not a source. 
Eleven sources are listed, and about 5 1/2% to 9% of the students do not 
respond. 

The HERI questionnaire asks the students to indicate the religious 
preference of self, father, and mother. From 15X to 17X do not answer the 
question. 

On the HERI questionnaire 6X of the entering freshmen do not Indicate 
their probable undergraduate major, and nearly 7X do not Indicate their 
probable career occupation. On tne ACT questionnaire the percent of omits 
1s 12X for the probable major and 16X for the probable occupation. The 
reason for these larger numbers may be owing to the format. The ACT survey 
has a separate sheet Inserted with the questionnaire listing many major 
fields and occupations. The student finds the 3-d1g1t code that best 
describes his plans and then fills 1n these numbers on the questionnaire. 
Apparently some students just don't bother to do this. On the HERI 
questionnaire the various fields and occupations are listed on the 
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queslonnaire Itself, making the response easier to record. In both cases, 
however, 1t seems reasonable to suspect that asking entering freshmen about 
their probable college major and their probable occupation Is not viewed as 
an answerable question by some students. In fact, on a different part of 
the HERI questionnaire more than 201 of the students said the chances were 
very good that they would change their major and change their occupational 
choice. 

In both questionnaires, Items about such topics as reasons for going 
to college and reasons for going to this particular college, were omitted 
by only 2X to 4X of the respondents In most Instances. The ACT 
questionnaire has a section labeled "college Impressions" where students 
are asked to indicate their agreement with various statements about the 
college environment — such as "students at this college are friendly", 
"this college offers many cultural events and programs". Typically about 
31 omit these Items; although one wonders about the basis for the answers 
because often one's Impressions, In advance of actual experience, reflect 
common stereotypes about what college Is like. 

The HERI questionnaire asks students questions about various 
political, social, and educational policies — such as "abortion should be 
legalized", "college grades should be abolished", "the federal government 
Is not doing enough to control environmental pollution". Typically about 
5Z to 82 of the students do not answer these questions. Another question 
asks students to characterize their political views, as far left, liberal, 
middle-of-the-road, conservative, or far right. About 5X do not answer the 
question. 
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For all of the above data, the Information about the ACT questionnaire 
comes from a normative report based on about 16,000 cases 1n which the 
number of "blank" responses to every Item 1s listed. For the HERI 
questionnaire the data come from the 1983 report of freshmen nor:ns 1n which 
the data for one sample college are shown, having about 2,300 cases. The 
complete normative report does not show missing cases. 

Except for the questions about major field and probable occupation, 
the number of "omits" In th- ACT questionnaire 1s generally smaller tf«an 1n 
the HERI questionnaire. There may be several factors accounting for 
this. Tne ACT questionnaire 1s shorter. The forma', and organization are 
also clearer. Section 1 Is labeled Background Information, Section 2 1s 
Educational Plans and Preferences, Section 3 1s College Imprerlons. 
Although 1n some parts the print 1s quite small, each part 1s enclosed In a 
box, w1tn the question or topic Itself 1n boldface capital letters. 
Perhaps more Important 1s the likelihood that most students would not view 
any of the questions as offensive or Intrusive. There 1s no Invasion of 
privacy of the sort that might Influence one to omit the answer or to give 
a socially desirable answer rather than a more forthright answer. One can 
easily regard the questions as appropriate to ask of entering students 
because of the educational relevance of the questions. 

The HERI questionnaire, although of the same four-page length as the 
ACT, has many more Items, and the format consequently appears crowded. 
Also there 1s no obvious organization or sequence to Its questions. The 
reasons for not answering various questions, however, are probably owing 
more to the nature of the questions than to the format. Questions about 
the future — such as "what 1s your best guess that you will": graduate 
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with hon rs, change career cho L :e, transfer to another college, find a job 
after college 1n the field tor which you were trained, etc.? are 
generally skipped by 51 to 61 of the respondents. Questions about 
longrangt aspirations or values are skipped by 5X to 8X of the students. 
Also, as noted earlier, questions about political and personal attitudes 
are typically skipped by 5X to 8X of the students. From one perspective, 
these are not large percentages; and the conclusions one draws from those 
who do respond would not be changed 1n any significant way 1f everyone had 
responded. From another perspective, these percentages of missing cases 
may represent the tip of a deeper and larger problem atuut the validity of 
students responses. There 1s no doubt that some students do not like some 
of the questions. During the time of student activism 1n the late 1960s, 
there were organized student protests against answering the sort of 
questions that are still Included 1n todays edition. At the end of the 
questionnaire, 26X of trie students do not give permission to Include their 
ID number on any tape for future research or follow-up study. This 
undermines the validity of the data base for longitudinal studies. 
Moreover, when one realizes that the response rate to a mailed foPow-up 
questionnaire may be only 50% or even less, then, together with the 26% 
refusal to be Involve*!, one 1s left with a respondent population that may 
be onl> 1/3 or 1/4 of the population one should have. 

Missing cases have also been noted for a third Instrument Pace's 
College Student Experience questionnaire. Later 1n this report a detailed 
examination of the reliability and val1d1t of responses will be 
presented. At this point, only the data about missing cases are reported. 
Most of the questionnaire consists of 142 college activities to which the 
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students respond by Indicating whether and how often they have engaged 1n 
them during the current school year. These are, for the most part, quite 
specific events, and apparently quite easy to recall. Based on the 
responses of about 7,500 undergraduates, the number of missing responses 
was rarely more than IX, and never more than 2X. These activities are 
grouped <nto scales, usually of 10 Items, to which an activity score can be 
computed. If any Item 1n a scale 1s not answered no score 1s computed. 
The number of missing cases 1n these scale scores 1s, 1n most scales, about 
2X or less, and never more than 4X. In other parts of the questionnaire 
students are asked to Indicate how much progress they believe they have 
made with respect to various goals or objectives, how well satisfied they 
are with college, and how they would characterize the college environment 
along various dimensions. The missing cases to these Items are often fewer 
than IX and never more than 2X. In another brief section of the 
questionnaire students are asked to Indicate about how many textbooks they 
read, how many non-assigned books, how many essay exams they had, and how 
many other written reports they made during the current school year. The 
percent of missing responses was typically from less than IX to 2X, except 
among students 1n not highly selective liberal arts colleges where there 
were 3X to 4X missing ca»es. No obvious explanation comes to mind for 
these somewhat larger percentages. With respect to the usual background 
Items — age, sex, year 1n school, etc. — there are typically no more than 
IX or 2X missing cases, except for the questions about the student's major 
field where the percent of missing cases ranges from 3X to 6X at different 
types of Institutions. Unlike the ACT and HERI questionnaires which are 
given to beginning freshmen, the CSEQ 1s answered by undergraduates 1n 
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general, not just by freshmen, so that most of them do 1n fact have a 
definite major field. Why there should be from 31 to 6X omits 1s a 
mystery. Of course, not all possible major fields can be listed In the 
questionnaire so that students may wonder where to classify their own 
major. Also, especially 1n the more heterogeneous colleges, and also 1n 
the most selective ones, there may be more Interdepartmental majors or 
other special options. Apparently, Instead of checking "other" as the 
proper response, they just omit the Item. 

The College Student Experiences Questionnaire: A Brief Description . 

To understand some of the analyses to be reported next, some knowledge 
about the content of this questionnaire may be helpful. The questionnaire 
1s meant to be filled out by undergraduates toward the end of the academic 
year. It 1s an eight page, 8 1/2 by 11 format, with the cover page 
Indicating what Its all about, and stating that "we do not ask you to write 
your name anywhere 1n this questionnaire; but we do need to know where the 
reports come from, and that 1s why each questionnaire has a number on the 
back page — certain blocks of numbers tell us that those questionnaires 
come from your college". The first 1 1/2 pages consist of "Background 
Information" — the usual questions about age, sex, year 1n school, college 
residence, major field, parents education; and also time spent on academic 
work, time on a job, main source of funding for college, grades, race, and 
citizenship. The next 3 1/2 pages are labeled "College Activities". There 
are 142 activities, grouped Into "scales" or topics labeled library 
experiences, experiences with faculty, course learning, art-mus1c-theater, 
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student union, athletic and recreation facilities, clubs and organizations, 
experiences In writing, personal experiences, student acquaintances, 
science/technology, dcrmatory or fraternity/sorority, topics of 
conversation, and Information 1n conversations. The directions are: "In 
your experience at this college during the c v .it school year , about how 
often have you done each of the following?" The responses are "never", 
"occasionally", "often", and "very often". The activities are fairly 
specific so that the student would presumably recall accurately whether he 
had ever done them; but of course the frequency estimate 1s entirely a 
suojectlve response. Examples of activities are as follows: read 
something 1n the reserve book room or reference section, made an 
appointment to meet with a faculty member 1n his/her office, summarized 
major points and Information 1n readings or notes, gone to an art gallery 
or art exhibit on the campus, meet yoLr friends at the student union or 
student center, played on an Intramural team, worked on a committee, asked 
other people to read something you wrote to see 1f 1t was clear to them, 
sought out a friend to help you with a personal problem, rcide friends with 
students from another country, practiced to Improve your skill 1n using 
some laboratory equipment, gone out with other students for late night 
snacks, talked about curre.it events 1n the news, referred to something a 
professor said about the (conversation) topic. 

The next brief part of the questionnaire asks students to report how 
much reading and writing they hav • done, and how well satisfied they are 
with col leg' 

The next main topic 1s labeled "The College Environment". This 
consists of eight rating scales on which students repor their Impressions 
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of the emphasis or stress there Is In the environment on sach aspects of 
students' development as academic and scholarly qualities, esthetic and 
creative qualities, being critical and analytical, vocational and 
occupational competence, and the general relevance and practical values of 
the courses; also their Impressions of the personal relationships In the 
environment, ranging from supportive, helpful, cons1dera.e to alienated, 
unsympathetic, and rigid with respect to the relationships among students, 
between students and faculty, and with administrative personnel. Finally, 
the last section, labeled "Estimate of Gains", lists 21 goals or objectives 
of college education and asks students as follows: "In thinking over 
your experiences In college up to now, to wh*t extent do you feel you have 
gained or made progress In each of the following respects?" The responses 
are "very little", "some", "quite a bit", and "very much". Here again, the 
responses are entirely subjective. 

From one perspective, this College Student Experiences que.clonnalre 
Includes features that some think should be avoided. If possible. The 
ratings are entirely subjective, the estimates of the amount of gain are 
entirely subjective, and the reports of frequency of participation In 
activities are entirely subjective. What follows next are examples of how 
subjective responses can be objectively validated. 

Test-Retest Comparisons 

In the absence of any major changes In the campus environment or 
facilities or admissions policy, one would expect some consistency In the 
amount, scope, and quality of effort revealed by students' responses to the 
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activities scales by different but comparable samples. Recently, several 
colleges have used the College Student Experiences questionnaire twice — 
once 1n 1984, and again 1n 1985. Such comparisons are not, strictly 
speaklno, an Indication of the reliability of self-reports. The answers 
come from different students and from a different time. Some changes 1n 
the responses may reflect true changes, not random changes or errors of 
measurement. Nevertheless, 1f one found substantial variations 1n the 
responses of two different but similarly selected samples, even though a 
year apart, one would worry about the dependability of the results. 

The best test-retest example comes from Denver University. It 1s best 
1n the sense that the sample size was fairly large ~ 635 1n the spring of 
1984 and 661 1n the spring of 1985. The samples were selected 1n the same 
way, the response rate was similar, and the two goups did not differ 1n 
such population descriptors as age, sex, year 1n school, major field, 
grades, residence, transfers, etc.. Nc attempt 1s made to compare the 
responses to every Item 1n the questionnaire. Rather, to get a general 
1nd1cat1un of stal*11ty, comparisons are made between the mean scores 
on each of the 14 activity scales, and the mean ratings on each of the 
environmental characteristics. It 1s not approprla' to report the scores 
on these matters, but 1t 1s permissible to report th* differences between 
the 1984 scores and the 1985 scores. The second test-retest example comes 
from Case Western Reserve University — with a sample of 779 students 1n 
the spring of 1S84 and of 376 1n the spring of 1985. The characteristics 
of the two samples are nearly Identical with respect to age, sex, year 1n 
school, transfer status, major field, etc . The third example comes from 
Keuka, a small college for women 1n upstate New York — with 148 1n the 
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1984 sample and 130 In the 1985 sample, The groups were similar 1n al i 
respects except one: the 1985 sample had a larger proportion of freshmen. 

On the l0-1tem activity scales the possible range of scores 1s 30 
points; 36 points on the three 12-1 tern scales; and 20 points on the one 
6-item scale. The typical standard deviations are 5*7 on the l0-1tem 
scales, 6.0 on the l2-1tem scales, and 3.2 on the 6-1tem scale. A glance 
at the 11st of differences 1n the table quickly reveals that at all three 
schools the magnitude of differences 1s usually less than one point. This 
1s true of 13 out of 14 scales at Denver, all 14 at Case Western Reserve, 
and 10 of the 14 scales at Keuka. In fact, at Denver the difference 1n 
mean scores between the 1984 and 1985 samples 1s .5 or less on 10 of the 
scales, at Case Western Reserve the differences are .5 or less on 13 of the 
14 scales; anu at Keuka on 6 of the 14 scales. 

If comparable scores from comparable samples, even though a year 
apart, 1s an Indication of test reliability, then obviously these student 
self-reports are very stable and dependable. At Denver, where there 1s a 
significant difference of 2.4 points on the Student Union scale, the 
explanation 1s a good example of a change 1n results owing to a change 1n 
conditions. During 1984 at Denver a new student union and activity center 
was under construction; 1985 was the first full year of Its operation, and, 
not suprl singly, the activity score for students 1 use of the union 
Increased significantly. At Keuka the differences between mean scores, 
although greater than 1.00 on four of the scales, are not statistically 
significant. N 

From these comparisons, one can surely conclude that self-reported 
activities and self-reported ratings of environmental characteristics are 
dependable and consistent. 
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Test-Retest Mean Differences — 1984 and 1985 
In Activity Scale Scores and Environment Ratings 



Activity Scales 


Denver 
Univ. 


Case Western 
Reserve 


Keuka 
College 


Library Experiences 


.3 


.4 


.7 


Experiences with Faculty 


.3 


.1 


1.3 


Course Learning 


.6 


.2 


0 


Art, Music, Theater 


.5 


.4 


.1 


Student Union 


2.4 


.2 


.4 


Clubs and Organizations 


.5 


.9 


.5 


Athletic and Recreation 


.1 


.4 


0 


Experience 1n Writing 


.5 


.2 


.2 


Personal Experiences 


0 


.3 


.7 


Student Acquaintances 


.6 


.1 


1.4 


Science/Technology 


0 


.■5 


1.2 


Donnatory or Fraternity/Sorority 


.9 


0 


l.S 


Topics of Conversation 


.5 


.4 


.6 


Information 1n Conversations 


.1 


.2 


.8 


Environment ratings 








Academic 


.1 


0 


.1 


Esthetic 


.1 


.2 


.2 


in Li ca i / ana ly v» ca i 


.1 


•t 

ml 


1 

.1 


Vocational 


.2 


0 


.2 


Personal Relevance 


.2 


.2 


.1 


Students 


.3 


0 


.1 


Facul ty 


1.3 


0 


.2 


Administration 


.1 


0 


.4 
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External Validity: Self-reported gains vs objectively known achievement 

Over the past 50 years hundreds of thousands of college students have 
taken objective achievement tests 1n various college subjects* tests 
constructed by national testing agencies. Certain conclusions from all 
this testing are so consistent, and so obvious, that 1t almost seems 
unnecessary to state them; but 1f one 1s to document that self-reported 
achievement corresponds to objectively tested achievement, then some 
examples of the test evidence must be given. The examples that follow are 
reported 1n Pace, Measuring Outcomes of College , Jossey-Bass, 1979. 

The first example shows the relationship between credit hours and test 
scores from the Pennsylvania study 1n 1928. The obvious elusion 1s that 
students learn what they study, and the more they study the more they 
learn. Students who had the most credit hours 1n the natural sciences had 
the highest test scores on the natural sciences test Items. The same 1s 
true tor credits and scores 1n language, literature and fine arts, and also 
for credits and scores 1n social studies. 
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Credit Hours and Tost Scores: 4500 Seniors from 
45 Colleges In Pennsylvania, Tested 1n 1928 



Natural Sciences Credits 
High: 55 or more 
Statewide average: 37 
Low: 6 or fewer 

Language, Literature, and Fine 
Arts Credits 

High: 67 or more 

Statewide average: 42 

Low: 12 or fewer 



Social Studies Credit Hours 
High: 97 or more 
Statewide average: 52 
Low: 12 or fewer 



Natural Sciences Test Scores 
120 
78 
46 



Language, Literature or 
Fine Arts Test Scores 

250 

168 

111 



Social Studies Test Scores 
292 
241 
196 
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The second example, some forty years later, comes from the Area tests 
of ETS's Undergraduate Assessment Program. The test results are based on 
47,000 seniors from 211 colleges In the years 1969-1971. For each of the 
three Area tests — Humanities, Natural Sciences, and Social Sciences — 
the average score for all seniors Is compared with the average score of 
seniors whose "area of Interest" corresponds to the subject matter of the 
test. The scores are standardized scores In which the standard deviation 
1s 100 points. In the humanities area the differences between the two 
groups Is 55 points. In the natural sciences area the differences are 57 
points and 66 points. In the social sciences area the difference Is 2 
points. The sub-group Is also part of the total group; and since 601 of 
the total group of seniors are also In the social science Interest group, 
the difference Is necessarily small. 
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UAP Area Tests: Approximately 47,000 Seniors from 
211 Colleges 1n the Years 1969-1971 



Humanities Scores 
All seniors 470 

Seniors whose area of 
Interest is 1n humanities 
(211 of all seniors) 525 



Natural Sciences 
Test Scores 

All seniors 430 

Seniors whose area of 
Interest 1s 1n biological 
sciences (12X of all seniors) 537 

Seniors whose area of 
Interest 1s 1n physical 
sciences (71 of all seniors) 556 



Social Sciences 
Test Scores 

All seniors 446 

Seniors whose arsa of 
Interest 1s 1n social 
sciences (60S of all seniors) 448 
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The third example comes from the UAP tests 1n designated major fields 
rather than from the more general Area tests. These results are shown 1n 
relation to the number of courses students had taken 1n their major — 
fewer than eight vs eight or more courses. It 1s unlikely that , 1n one's 
major field, one would have fewer than six courses and still qualify as a 
major. Most likely, the comparisons are between students who have had 6 or 
7 courses vs those who have had 8 to 12 courses. Again, the more one 
studies a subject the mere one knows about it. 

Given these obvious conclusions from decades of achievement testing, 
one can surely use them as external validity 1n relation to self-reported 
achievement. The College Student Experiences questionnaire, 1n the section 
labeled Estimate of Gains, lists 21 objectives. Students are asked "to 
what extent do you feel you have gained or made progress 1n each ...?" 
They could check "very little", "some", "quite a bit", or "very much". Not 
all of the objectives are associated with a specific major field, or even 
with any course-related experience ~ objectives such as "ability to 
fuctlon as a team member", "ability to learn on your own, pursue ideas, and 
find Information you need". There are, however, eight goals that are 
related to the curriculum, and specifically to a major field within the 
curriculum, or to a specific type of subject-matter. These subject-matter 
goals Include Fine Arts, Literature, English (writing), Science, 
Technology, Computers, Quantitative thinking, ana Ph1losophy/Cui tures. If 
student self-reports are valid they should show the same results that test 
scores show — higher achievement (progress) by students whose major field 
1l similar to the objectives as compared with the average of all students 
— and this 1s exactly what the results do, 1n fact, very clearly show. 
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Scale Scores of Seniors on Major Field Tests of the Undergraduate 
Assessment-Program, 1969-1971, Related to Number of Courses Taken 

1n the Major Field 



Sciences a.<d Engineering Tests 
Biology 
Chemistry 
Engineering 



Fewer than Eight or 
eight courses more courses Difference 



539 
510 
506 



566 
53^ 
528 



+ 27 
+ 2? 
+ 22 



Humanities Tests 
History 
Literature 
Philosophy 
French 



468 
455 
514 
448 



491 
491 
551 
486 



+ 23 
+ 36 
+ 37 
+ 38 



Note: The number of students tested varies by major field, ranging 
from approximately 1,000 to 8,000. 



83 



38 



The data presented here are from a composite of 13,650 undergraduates 
from 49 colleges and universities who responded to the CSEQ 1n the spring 
of 1983, 1984 or 1985. Only those colleges that had given the 
questionnaire to a 1 four undergraduate classes are Included 1n these 
composite results. Note also that knowledge or progress 1s necessarily 
less among freshmen or sophmores who have not yet accumulated many credits 
In what 1s or will be their major field, than 1t would be among juniors and 
seniors who, by definition, have accumulated a much larger number of 
credits in their chosen major field. For some, then, the "major" may 
reflect an "area of Interest" and for others 1t may be a cou se program 
nearly completed. 

In the 11st below, the first four goals are related to the subject 
im* *r of arts and humanities, and the second four goals are related to the 
sciences. Among student* who Identified their m*jor field as "Arts (art 
music, theater, etc.)", 92X reported substantial gain ("quite a bit" plus 
"very much") toward the objective "developing an understanding a»'d 
enjoyment of art, music and drama". This hv,i. percentage contrasts with 
29X among students 1n general. For the objective related to literature, 
74X of humanities majors reported substantial gain compared with 33X for 
students in general. With respect to writing clearly and effectively, 80X 
of the humanities majors reported substantial progress compared with 57X of 
students in general. The goal described as "becomlnc aware of different 
philosophies, cultures, and ways of life" 1s not so clearly tied to 
classroom «ubj matter 1n the sense that students' Interpersonal 
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experiences might well contribute significantly toward Its attainment; but 
presumably courses 1n philosophy, history, anthropology, etc. would also be 
Influential. The results show substantial progress reported by 7M of 
humanities majors, and 64X of social sciences majors, compared with 51X by 
students 1n general. 
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Comparisons of Self-Reported Gains 
with Known Data About Achievement 



Gains 1n developing an understanding and enjoyment of art, music, and drama 



ARTS majors reporting substantial gains 92% 

average of all students 29% 

Gains in broadening your acquaintance and enjoyment of literature 

HUMANITIES majors reporting substantial gains 74X 

average of all students 38X 

Gains 1n writing clearly and effectively 

HUMANITIES majors reporting substantial glans 80X 

average of all students 5"£ 

Gains 1n becoming aware of different philosophies and cultures 

HUMANITIES majors reporting substantial gains 70X 

SOCIAL SCIENCE majors reporting substantial gains 642 

averge *f all students SIX 

Gains 1n understanding the nature of science and experimentation 

BIOLOGICAL SCIENCES majors reporting substantial gains 85X 

PHYSICAL SCIENCES majors reporting substantial gains 76X 

average of all students 36X 
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Gains in understanding new scientific and technical developments 

BIOLOGICAL SCIENCES majors reporting substantial gains 742 

PHYSICAL SCIENCES majors reporting substantial gains 662 

ENGINEERING majors reporting substantial gains 66X 

average of all students 31X 

Gains 1n acquiring familiarity with the use of computers 

COMPUTER SCIENCE majors reporting substantial gains 90Z 

ENGINEERING majors reporting substantial gains 651 

average of all students 32X 

Gains 1n quantitative thinking — understanding probabilities, proportions, 
etc. 

PHYSICAL SCIENCES majors reportir * substantial gains 68Z 

ENGINEERING majors reporting substantial gains 68X 

average of all students 47X 
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In "understanding the nature of science and experiment? t1 on", 
substantial progress 1s claimed by 852 of biological sciences majors and 
762 of physical sciences majors, compared with 362 for students 1n 
general. A similar result 1s shown for "understanding new scientific and 
technical developments", with percentages of 742 and 662 for scientific and 
technical majors, compared with 312 for the average of all students. The 
contrasting percentages for the goal "acquiring familiarity with the use of 
computers" are even sharper — 902 of majors 1n computer science Indicating 
substantial progress compared with 322 for the average of all students. 
With respect to quantitative thinking, students majoring 1n fields where 
much quantitative thinking 1s required — engineering, and physical 
sciences — are most likely to claim substantial progress (682) compared 
with 472 among students 1n general. 

All of the above results document the external validity of students 
set. -reports. When asked to rate their progress toward goals that are 
obviously related to the subject matter of college courses, the ratings are 
totally congruent with what we know from achievement test scores and from 
the relationship between credit hours or amount of study and measued 
achievement. 

One does not know the actual level of measured achievement 
(standardized test scores) that 1s associated with the students' 
self-estimate of gain. No doubt some stidents who rate their own progress 
as "quite a bit" may have higher achievement test scores than students at 
another college who rate their progress as "very much". Such discrepancies 
probably reflect Institutional differences In academic selectivity and 
academic de*nands. The same variability applies to credit hours vs test 
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scores. While 1t Is true that the more courses one takes 1n a subject the 
more one 1s likely to know about It, It Is also true that some students who 
have taken 5 or 6 courses may get higher test scores than some students who 
have taken 9 or 10 courses. But the averages are consistent. Sorting 
students according to course work (major field) or according to achievement 
test scores (major field) or according to self-reported progress On major 
fields) all produce the same conclusions. 

Internal Reliability; Consistency in responses to similar Item s 



In the Science/Technology activity scale there are three activities 
th*t clearly Involve conversation about science. These items, together 
with the percent o. students who said they er"~?ed In them frequently, are 
shown below: 



Science activities 



% Frequently among 
Blo.Sd . Phys.Sd . Engr , 
majors majors majors 



Tested your understanding of 70 
some scientific principle by 
seeing 1f you could explain 
1t to mother student. 

Showed a c 1 assmate how to use 43 
a piece of scientific 
equipment 

Attempted to explain an 43 
experimental procedure to 
student 

Conversation topic 

Science — theories, experl- 57 
ments, methods 



69 



35 



42 



r* 1 



69 



34 



41 



Average 
of all 
students 



34 



18 



15 



21 
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The conversation Item appears 1n a different part cf the 
questionnaire. Presumably, the percent of students who say they have 
frequently talked about science with other students should have some 
similarity to the percent who said they had tried to explain a prlirlple. a 
procedure, and the use of equipment to another rtudent. The responses 
were, 1n fact, very similar. 

A similar comparison can be made 1n the arts. In the activity scale 
labeled Art, Music, Theater there are three "talk about" Items, and later, 
among v,e conversation topics there 1s a topic described as "Fine arts - 
painting, theatrical productions, ballet, symphony, etc.". Here are the 
results. 

Among 



Art, Music, Theater 
activities 



Talked about art (painting, 
sculpture, architecture, 
artists, etc.) with other 
students at the college 

Talked about music (classical, 
popular, musicians, etc.) with 
other students at the college 

Talked about the theater (plays, 
musicals, dance, etc.) with 
other students at the college 

Conversation topic 

fine arts — painting, theatrical 
productions, ballet, symphony, etc. 



% Frequently among 
Arts majors 



68 



73 



58 



78 



all 
students 



17 



35 



20 



17 



Similar but not Identical questions produce similar but not Identical 
answers. The general congruence shown above can be regarded as an 
indlcallor. of Internal reliability. 
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Internal Validity: finding plausible connections 

For the attainment of many goals of higher education there is no 
readily available objective documentation and in some cases no external 
evidence at all. One can use tests and credit when the goals are related 
to the curriculum or to particular courses and major fields. But what does 
one use for an external criterion when the goals are self-understanding, 
understanding others, good health habits, functioning as a team member, 
etc.? 

In this part of the report several examples of internal consistencies 
that should be found are used to make the case for the credibility of 
self-reports. The first example is surely a connection that should exist. 
The activities — setting performance goals, fo T1 owing a regular exercise 
schedule, and keeping a record of progress — are, to a considerable 
extent, behavioral Indicators of what 1s involved 1n "developing good 
health habits and physical fitness". The tabulations show that students 
who report "very much" progress toward this goal are much more likely to 
set goals, follow a schedule, and keep a record than students whose 
self-rated progress is lower. 

Similar tabulations are s r n for several other goals. In every case, 
the behavior that surely should contribute to students 1 estimated progress 
1s clearly related to that progress. The differences in percents between 
"very much" and "very little" are uniformly large, the one being from 2 to 
mor*e than 6 times larger than the other. 

If student responses to the gains items or to the activity items were 
capricious or unreliable or Invalid, the congruent and plausible 
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connections shown In the tables below would not occur. If what should be 
true Is also true empirically, the credibility of self-reports Is further 
documented. 
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Goal: Developing good health habits and physical fitness 



Percent engaging 1n activity 
frequently among students who 
rate their progress as: 



Activity 


Very 
Much 


Quite 
a bit 


Some 


Very 
L'ttle 


Average of 
all students 


Set goals for your performance 
1n some skill (athletic) 


77 


58 


36 


23 


45 


Followed a regular schedule of 
exercise, or practice 1n some 
sport, on campus 


71 


53 


28 


14 


3f 


Kept a chart or record of your 
progress 1n some skill or 
athletic activity. 


28 


15 


6 


3 


11 



Goal: Ability to function as a team member 



Percent engaging 1n activity 
frequently among students who 
rate their progress as: 



Activity 


Very 
Much 


Quite 
a bit 


Some 


Very 
Little 


Average of 
all student* 


Used outdoor recreational spaces 
for casual and Informal group 
sports 


40 


27 


15 


7 


23 


Used facilities 1n the gym for 
playing sports that require 
more than one person 


42 


30 


18 


10 


26 


Played on an intramural team 


36 


26 


15 


7 


22 
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Goal: Understanding yourself — your abilities, Interests, and personality 



Percent engaging 1n activity 
frequently among students who 
rate their progress as: 



Activity 


Very 
Much 


Quite 
a bit 


Some 


Very 
Little 


Average of 
all students 


Read articles or books about 
personal adjustment and 
personality development 


38 


25 


20 


15 


28 


Asked a friend to tell you 
what he/she really thought 
about you 


33 


21 


14 


12 


23 


Identified with a character 1n 
a book or movie and wondered 
what you might have done under 
similar circumstances 


56 


44 


36 


32 


46 



Goal: Understanding other people and the ab1lt<y to get along with 
different kinds of people 



Percent engaging 1n activity 
frequently among students who 
rate their progress as: 



Activity 


Very 
Much 


Quite 
a bit 


Some 


Very 
Little 


Average of 
all students 


Made friends with students 
whose Interests were very 
different from yours 


73 


57 


38 


32 


59 


Made friends with students whose 
family background (economic 
and social) was very different 
from yours 


78 


63 


44 


36 


63 


Had ser'ous discussions with 
students whose political 
opinions were very different 
from yours 


45 


33 


26 


22 


35 
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Goal: Becoming aware of different philosophies, cultues, and ways of life 



Percent engaging 1n activity 
frequently among students who 
rate their progress as: 



Activity 


Very 
Much 


Quite 
a bit 


Some 


Very 
Little 


Average of 
all students 


Made friends with students 
whose race was different 
from yours 


62 


50 


40 


33 


46 


Made friends with students 
from another country 


50 


24 


24 


20 


31 


Had serious discussions with 
students whoso philosophy 
of life or personal values 
were very different from yours 


64 


48 


33 


25 


43 


Had serious discussions wit 1 ) 
students whose religious 
beliefs were very dlffrent 
from yours 


55 


40 


28 


22 


36 


Had serious discussions w1:h 
students from a country 
different from yours 


42 


25 


15 


13 


2? 
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Summation 

Claims for the credibility of student self-reports can be supported 

by: 

1. Evidence of test-retest consistency. 

2. Congruence with externally known facts, when such facts are 
available. 

3. Similar answers to questionnaire items, when questions are 
asked in more than o*,<* way. 

4. Congruent, or expected, connections between items that 
presumably should have connected responses ~ as for example between 
behavior and progress. 

One final note may be important. Some psychometrlcians and survey 
research analysts point out that the context within which questions are 
asked may influence the response. In the College Student Experiences 
questionnaire, some people might claim that the answer to the Estimates of 
Gains Items might be "contaminated" by all the Dreceding Items. The gains 
might be reported differently 1f they were asked separately, or without the 
prior contexl in the questionnaire. There is, however, a very different 
way of regarding this matter. If the gains items were presented alone, 
without sny context, the responses would be all the more influenced b w 
personal Idiosyncrasies, and hence all the more likely to produce random 
variations. By putting the gains 1w*ms at the end of the questionnaire, 
one increases the credibility of answers. Everyone comes to these items 
with the same background, having recalled one's behavior during the year, 
having characterized the college environment, having reported how much o».i 
has studied, what grades one has received, etc. so that, for everyone, tne 
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estimate of gains becomes a more or less commonly based and thoughtful 
summary of the college experience, and therefore has a greater reliability. 
Finally, as a capstone Illustration of what can be done to assess the 
reliability of self-reports, we apply some multivariate statistical 
procedures which bear upon the predictive and construct validity of certain 
parts of the Collie Student Experiences questlornaire. 

Multivariate statistical procedures 

In this part of the report we describe the use of common multivariate 
statistical procedures to assess the validity cr self report. The goal 1s 
to demonstrate that for surveys that allow internal validity checks, one 
can go beyond item-by-1tem validity to assessing the validity of self 
report at the construct level. These techniques are applied to a sample of 
6,000 undergraduates who provided responses to the College Student 
Experiences Questionnaire (CSEQ). 

Three techniques were applied tc two types of scales and one 
background variable of the CSEQ. The background variable 1s academic major 
coded as: 1) Arts; 2) 31olog1cal Sciences; 3) Business; 4) Computer 
Science; 5) Education; 6) Engineering; 7) Health related fields; 8) 
Humanities; 9) Physical Sciences; 10) Social Sciences. The two types of 
scales are composed of 13 subscales from the Quality of Effort (QE) 
measures, and 21 Items from the Estimate of Gains (EG) measures. 

The first statistical procedure 1s discriminant analysis ***h special 
attention paid to the classification phase of the analysis. The 
discriminating variables are the EG items while the classification variable 
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1s academic major. Since th» number of undergraduates 1n each major are 
not the same, special a p riori weighting 1s given to the samples during the 
classification phase. The rationale for using discriminant analysis 1n 
this context is that those who major 1n certain academic disciplines 
probably make the most gains in those areas related to that discipline. 
Hence, if one knows a student's set of responses to the gains Items, one 
should be able to predict that Individual 1 s major. To the extent that this 
1s true, 1t might be argued that the EG measures provide valid self report 
measures of gains. 

The second procedure 1s canonical correlation analysis applied to the 
QE subscales and the EG items. This procedure attempts to find a set of 
linear combinations (canonical varlates) within a scale that are maximally 
correlated with linear combinations formed from the other scale. To the 
extent that these canonical varlates are interpretable, we would expect 
high canonical correlations among those varlates from each set that have 
something in common. Often it 1s the case that canonical analysis obscures 
the simple factor structure that might exist within a set of items. To 
address this problem, the third procedure 1s to factor analyze the QE 
subscales and EG items separately, rotate the factors for maximum 
interpretabilty, calculate factor scores, and correlate factor scores using 
simple Pearson correlations. It 1r expected that Pearson correlations 
should show high correlations among those factors that are substantively 
related. 

Discriminant and classification analysis were performed using ten 
academic majors and twenty EG Items. The EG Items were chosen to 
correspond as closely as possible to the academic majors, hence the item 
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related to gains 1n vocational training v;as omitted since no major was 
uniquely vocational. 

The results of the classification phase are displayed 1n the following 
table. The table shows the .entage of those who were classified Into 
their known majors on the basis of the discriminant analysis. The 
dlagnonal represents the percentage of correct classifications, while the 
off-diagonal represents the m1sclass1f1cat1ons. It can be seen that the EG 
responses tend to do well 1n predicting academic major. For example, 611 
of all art majors were correctly classified as being art majors on the 
basis of the discriminant analysis. Certain Incorrect classifications did 
occur; but the m1sclass1f1cat1ons were 1n a sensible direction. For 
example, physical science majors (Including chemistry and math) were more 
often classified as biological science majors (Including biochemistry) or 
engineering majors. Overall, these results lend support to the claim that 
self report of gains as measured by the EG data are valid 1n that they 
adequately predict a relatively objective measure of academic f1e* 1 where 
most gains should occur. 

The results of the canonical analysis are displayed 1n the next 
table. Here, only the first two canonical varlates extracted from each 
set of measures are presented. Note that the standardized canonical 
coefficients can be loosely Interpreted as factor loadings. 

Inspection of the standardized canonical coefficients for the QE 
subscales suggests that the first canonical varlate 1s dominate by the QE 
subscale related to Science and Technology. The first canonical varlate to 
the EG data appears to be dominated by those Items related to computer 
knowlege and Science/Technology. The squared canonic?! correlation between 
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Classification Analysis of Academic Major 
on Basis of Discriminant Analysis* 



Predicted Major 



True Major 


Art 


B1o 


Bus 


C/S 


Ed 


Engr 


heal 


Hum 


PhyS 


SocS 


Total 


Arts 


61 


4 


11 


1 


0 


0 


C 


11 


0 


11 


100 


Bio 


1 


52 


7 


1 


0 


15 


13 


0 


0 


10 


100 


Bus 


2 


1 


73 


4 


* 
c 


3 


2 


1 


0 


12 


10C 


CompSd 


1 


1 


33 


45 


0 


9 


0 


1 


0 


1 


100 


Educ 


9 


2 


34 


2 


15 


3 


9 


7 


0 


18 


100 


Engr 


1 


10 


14 


8 


0 


57 


4 


0 


1 


5 


100 


Health 


2 


20 


15 


0 


4 


4 


34 


1 


0 


19 


100 


Human 


8 


3 


11 


1 


3 


1 


3 


38 


0 


32 


100 


Phs/Sc1 


0 


33 


17 


7 


1 


25 


5 


1 


3 


7 


100 


Soc/Scl 


4 


7 


28 


2 


4 


3 


4 


9 


0 


39 


100 


TOTAL % 


5 


11 


33 


6 


3 


11 


7 


6 


0 


17 


100 



* Entries are 1n percentages. 
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Standard Canonical Variates for 
QE Scale and EG Scale Items 



EQ Canonical 


Variates 


QE2 


EG Canonical 


Variates 


EG2 


Subscales 


QE1 


Items 


EG1 


Library 


.02 


.05 


Professional 


.03 


.10 








Sc1 or Scholarly 






Faculty 


- .06 


.10 


General Education 


- .02 - 


.02 


Course Work 


- .05 


.15 


Career Development 


- .07 


.03 


Art. Music 


- .16 


.66 


Art, Music, Drama 


- .14 


.64 


Drama 












Student Union 


.00 - 


.05 


Literature 


- .07 


.16 


Recreation 


- .00 - 


.05 


Writing 


- .17 - 


.13 


Clubs 


f\f\ 

.00 


.04 


computers 


.51 


fin 
.09 


Writing 


- .14 - 


.03 


Philosophies/ 


- .08 


.09 








Culture 






Personal 


- .14 - 


.03 


Ethical Standards 


- .02 


.08 


Experiences 












Acquaintances 


- .04 


.01 


Personality 


- .03 


.01 


Sc1/Tech. 


.94 


.21 


Understanding 


- .05 


.08 








People 






Conv. Topics 


.03 


.21 


Team Work 


.01 


.04 


Information 


- .06 


.20 


Physical Fitness 


- .03 - 


.06 








Science Experlm. 


.27 


m 
. Ul 








Science/Technology 


.32 


.15 








Technol ogy/Hazards 


.00 


.14 








Analytical Thinking 


.05 - 


.01 








Quantitative Thinking .16 - 


.15 








Similarities and 


- .11 


.20 



Differences 
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these two canonical varlates 1s statistically significant [ R 2 = 0.61, 
F (260, 44745) « 42.979, p < .000 ]. 

Inspection of the second canonical varlates for both sets of 
measures reveals a similar consistent picture. The second canonical 
variatc for the QE scale 1s related to art, music, and theater, while the 
second canonical variate of the EG responses 1s related to gains 1n 
understanding art, music, and drama. Again, the squared canonical 
correlation between this pair of variates 1s statistically significant 
[ R 2 = 0.38, F (228, 41681) * 29.011, p < .000]. Subsequent canonic^ 
varlates were difficult to Interpret. 

It can be seen that the canonical analysis gives a useful, though 
limited, picture of the Internal validity of the two self report measures. 
Again, 1t should be noted that this procedure examined validity of self 
report at the construct level, where the canonical varlates can be taken as 
representing the constructs, though perhaps not 1n the factor analytic 
sense. 

On the basis of previous research, four factors of the QE scale and 
five factors of the EG Items were Independently extracted and obliquely 
rotated to simple structure. The four factors of the QE scale were labelec 
1) Personal/Social; 2) Academic/Intellectual; 3) Clubs/Organizations; 
4) Science. The five EG factors were labeled 1) Personal /Social; 2) 
Science/Technology; 3) General Education; 4) Intellectual; 5) Vocational. 
A matrix of Pearson correlations among the factor scores obtained from the 
factor analysis 1s displayed 1n the next table. Although most of the 
correlations are large and significant;, those that are highest are among 
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Intercorrelations Among Factor Scores 



Estimated 
Gains 
Factors 



P/S 



S/T 



G/E 



Intel 



Voc 



Quality of Effort Factors 
P/S A/I C/0 Sc1 



.50 


.42 


.44 


.11 


.19 


.13 


.04 


.62 


.42 


.45 


.31 


.07 


.36 


.36 


.22 


.43 


.29 


.29 


.24 


.25 
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those factors that have something 1n common. For example, the gains 1n 
personal and social development factor 1s most highly correlated with QE 
factor measuring personal and social aspects such as student acqualntences, 
personal experiences, and topics of conversation. 

Two points can be made with regard to the above analyses. First, the 
application of multivariate statistical procedures for assessing broad 
construct validity of self report has potential. It should be pointed out 
however, that construct valldty 1n the factor analytic sense was only 
explained via the factor score correlations. Secondly, with respect to the 
CSEQ, and the QE scales and EG Items 1n particular, evidence does exist for 
claiming a certain degree of validity 1n these self report measures. The 
result of all three analyses present a picture of a questlonalre that 1s 
consistent with respect to self report predictive validity and self report 
construct validity. 
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CONCLUDING COMMENTS 



This report 1s obviously not a definitive document about the 
credibility of questionnaire survey responses by college students. It has 
aimed, nevertheless, to show that there are many ways to confirm the 
accuracy, reliability, and validity of student self-reports. It has also 
noted, from examples in higher education, and from examples 1n the larger 
area of public opinion polls and other general surveys, some of the common 
sources of measurement errors and errors of substance. 

In academic surveys the high proportion of students who do not reply 
to the questionnaires they have received 1s a most serious problem. One 
wonders whether rigorous follow-up messages would make a big difference, or 
whether the magnitude of the non-respondent problem reflects a deeper 
rejection of such Inquiries. Twenty years ago one couVd expect about two 
thirds of college students to respond to a questionnaire. Today, one is 
grateful if 501 respond. Times change. Nearly 50 years ago, in a study I 
directed of former university students, Including some who had graduated 
and some who had not, we got returns from 70S of those who received the 
questionnaire. The questionnaire was 52 pages long and took about two 
hours to answer. But that was before the Invention of television! (Pace, 
They Went to College , University of Mlnesota Press, 1941). 

My own belief 1s that the likelihood of good returns 1s enhanced by 
the recipients 1 opinions about the Importance of the topic, Its perceived 
relevance to one's experience, one's regard for the source of the Inquiry 
and the likely use or value of the results, the clarity of questions and 
the ease or confidence one has in answering them, and the overall 
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attractiveness of the design, format, typography, etc. of the Instrument. 
I also believe that unless these conditions are reasonably well met, even 
vigorous follow-up efforts will have little Influence on the response rate, 
and even when some Increase 1n response rate 1s achieved I would be 
skeptical about the Integrity of those added responses. 

Perhaps the second most common weakness 1n questionnaires by academic 
organizations 1s the Inclusion of questions that are quite likely to have 
unreliable or Invalid answers. These may be questions about vague 
concepts, questions about topics that students have not previously thought 
about, questions about values or life goals or future plans. Similar 
weaknesses are evident 1n public opinion polls that ask for opinions about 
ambiguous or undefined concepts such as national defense, foreign aid, 
national health, etc. The unfortunate consequence 1s that pollsters and 
public alike think that the results reflect public attitudes toward the 
matter, when 1n fact the topic 1s complex, can be phrased 1n a variety of 
proper ways, and all one has done 1s to tally answers to the particular 
question which is not well or uniformly Interpreted 1n the first place. 
Questions about future expectations can be very clear ~ for example, "Do 
you expect to have any (more) children?" But 1t 1s difficult to know just 
what 1s being measured or revealed by answers to questions that different 
people can Interpret 1n different ways. 

A final Issue 1s the use of single questions versus the use of scales 
or combinations of questions that can be added together to produce a score 
or index. Commercial agencies rely on single Items. Scale development 1s 
complex, time-consuming, and costly; and for public opinion polling 
agencies the presumed benefit 1s not worth the price. A scale 1s not 
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always better (more reliable and valid) than a single Item. In most 
academic surveys, however, the topics of Inquiry tend to be rather global 
rather than narrowly explicit* In these cases there is merit 1n thinking 
about questionnaire construction In ways somewhat similar to thinking about 
test construction. 

Whatever the topic of Inquiry, 1t may well be that one of the most 
Important elements to consider 1n writing the questions 1s the nature of 
judgment required to answer them. If the judgment or thought process 1s 
one of recall, 1s the thing or condition to be recalled clear and are the 
respondents able to recall accurately? If the judgment 1s one of 
comparison, is the base for the comparison clear and do the respondents 
have the experience or knowledge needed to make the comparison with 
reasonable confidence. If the judgment or thought process to answer the 
questions 1s one of generalizing or inferring, do the respondents 
understand what is to be generalized? Many survey questions would probably 
yield better answers if the writers always asked themselves such questions 
as: Does the respondent have the knowledge or experience to give a useful 
answer? Will different people interpret the queslton in the same way? 
Will the answer be accurate? What can I conclude or interpret from the 
answers to this question? 

The quality of questionnaire answers (reliability, validity, 
credibility) depends most of all on the quality of the questions. 
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ASSESSING INSTRUCTIONAL OUTCOMES 



Eva L. Baker, Center for the Study of Evaluation, 
University of California at Los Angeles 

*nd 

Harold F. O'Neil, Jr., University of Southern California 

This chapter is addressed to thr topic of assessing Instructional 
outcomes. It occupies, conceptually, an Interesting point 1n the 
consideration of Instructional technology. On the one hand, the hallmark 
of technology 1s Its repeatable utility based upon Us use of verified 
knowledge produced from research. Assessment 1s clearly a requirement to 
determine 1f one has a technology that works. On the other hand, in 
practice, the serious consideration of assessing educational outcomes is 
often overlooked 1n the excitement of exploring Innovation or in the 
day-to-day tedium of producing sufficient amounts of courseware or other 
instructional products on schedule and within budgetary constraints. 
Because of the lack of attention to the Issue of educational outcome 
assessment, measuring outcomes in the recent history of instructional 
design has been treated routinely, more as an historical obligation than as 
a tool integrally related to the Improvement of instructional 
effectiveness. For this reason, it 1s important to see that the 
measurement of Instructional outcomes has two critical functions, 1) it is 
both a means to assess how well the product, courseware, or ott.er 
technology performs, and 2) it 1s a mechanism to Intervene *r *nd to 
improve the process of Instructional design and development Itself. 
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Basic to the understanding of the assessment of instructional outcomes 
is the role of tests . Unfor\ inately, the term tests conjures up some of 
the least useful forms of assessment and restricts the instructional 
designer's view of the full range of information useful for making 
important inferences about the effects of learning. While basic 
understanding of testing is important, and will be treated in this chapter 
as well, it is not sufficient. It is more important to think broadly first 
about information needed to make instructional decisions, and secondly, 
about the inferences one can draw from such information to make decisions 
about the quality of instructional efforts. 

At the heart of both the information base and the inferencing process 
is the notion of validity, and it should be the overriding concern in the 
process of assessment. The formats of asses^ent, where they take on the 
coloration of typical tests or even look very different from the tests we 
have seen and taken in school, are at best secondary concerns. Our intent 
1s to raise the salience of assessment in the entire design and development 
process by identifying the c *1 attributes of valid information and 
inferences. Then we will J discussion of the various sorts of 

testing and other assessment options and consider their strengths and 
limitations against a fr» lework of validity. 

Measurement: The Basics 
Without deeply investigating the psychometric theory underlying 
measurement, an instructional designer can still treat the assessment issue 
seriously. A few straightforward points need to be reviewed. First, all 
measurement is Imprecise. Everything we infer is >actly that: inferencing 
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about learning that has occurred (or 1s potential) In the learner. As 
measurement begins to use some of the newer techniques in the blotechnlcal 
area, readings of magnetic fields, heat, and other electrical brain 
activity, then we may appear to be closer to direct measurement of 
learning. But since we are dealing with the mind, we will still remain In 
the land of inference and Inevitably be left to piece together what has 
actually been experienced by the learner. 

Second, a good deal of what is measured 1s inaccurate because we have 
chosen the wrong thing to measure. We may have chosen an approach 
inappropriate to the subject matter, chosen to measure performance 1n a 
particular way because of its practicality and convenience rather than for 
reasons related to the accuracy of assessment. So even 1f we were to 
Improve our precision, we would err by selecting, some of the time, the 
wrong matter to which to attend. 

Third, we must remember we are dealing with people, not plastics. 
People are dynamic; all change from second to second. The meanings they 
ascribe to events become successively refined and restructured with 
experience. They are blurry targets for precise metrics. As we all know, 
people not only change continuously but they differ from one another 
enormously. They have color preferences, various language facilities, and 
predispositions to certain subject matter content, for Instance. They also 
have very different perceptions of themselves as learners, of their 
abilities to succeed, and of the reasons they succeed and fail (see Welner, 
for example, 1979). Some are desperately anxious when they are given tests 
(O'Neil & Richardson, 1980), some worry about only one sort of test, Hke 
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essays or multiple choice, and some are relatively accepting of whatever 

tasks come their way. 

People also think in different ways. Their approaches differ not 
only as a function of the level of ignorance or expertise they have about a 
single subject, but their general background or world knowledge. They also 
approach problems very differently. One style is methodical and analytic; 
learners of this sort see the world in terms of components that get built 
up or decomposed into smaller parts. Other see the world in broad 
patterns, seek integration, use metaphors, and 'ocus on the whole rather 
than its parts. And many people -se both approaches described, switching 
within the same problem sometimes to understand through one or another 
means. These approaches were described simply and archetypally to make a 
point. But, it should be remembered that a good deal of style of learning 
comes automatically to the learner. Only infrequently is learning style a 
volitional matter, although there have been moderately successful attempts 
to affect the use of various learning strategies (O'Neil, 1979; Danseresu 
et al, In Press; Moore et al, 1985). 

Our primitive measurement tools will miss a good deal of this 
complexity. So even if we had precise methods, and were confident that we 
were assessing the correct type of learning, we would still be sure to miss 
a good deal of the truths of what our effects have been. 

It is for all these reasons that we can not claim to have proved that 
our instruction is effective, just as we cannot prove. that a scientific 
theory is right. We have to repeat ou»- measurements, find multiple 
approaches to assess the outcomes we are intending, and still couch our 
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conclusions tentatively. In the educational marketplace, of course, 
tentativeness goes by the board. Instructional designers compete with 
claims about materials proven effective, quality assurance and other 
slogans designed to loosen resources from program managers either in 
business or in government. But, in the secret recesses of one's own mind, 
it is important to know what we don't know, even if our roles or 
organizations require different public proclamations. 
Purposes of Assessment 

Central to the problem of assessing Instructional outcomes 1s the 
issue of purpse: for what purpose are we to assess outcomes? One common 
enough response 1s to assess the quality of our intervention 1n meeting Its 
particular goals. If a program or system 1s devoted to teaching reading 
comprehension, then it 1s appropriate to assess the extent to which reading 
comprehension ability is affected by exposure to the Intervention. A 
second purpose of assessment in instruction?! contexts related to the 
Improvident of the program itself. We wish to assess instructional 
outcomes, again, reading comprehension 1n the example just given, for the 
purpose of revising instructional processes 1n the desired direction. 
These two purposes of assessment Interact, often sharing the same sets of 
data collection processes and measures. 

With both these outcome assessment purposes, the principal focus has 
been on the achievement produced by the Intervention, what and how well 
students learn, or to flip the perspective, how well the Intervention 
taught as a measure of Its effectiveness. Recently, the focus of outcome 
assessment has been broadened 1n a number of ways: 1) to assess both 
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cognitive and af Active outcomes other than those intended by the 
intervention; 2) to include measures of attitudinal development and 
satisfaction; 3) to assess how students go about learning, their processes 
rather than their products. An additional but largely unsatisfied quest is 
to determine for which students, based on student individual differences 
such as cognitive preference, experience, and ability, various 
instructional interventions are most effective (Cronbach and Snow. 1977; 
Clark 1983). 

But a critical focus is on the assessment of learning outcomes. The 
means to accomplish such assessment has been criterion-refererced 
measurement (CRM), and that is the major focus of this chapter. 

Criterion-Referenced Measurement - Some Background 
Criterion-referenced measurement has had many definitions. The merits 
of each and implications of different wording will iater be discussed at 
some length. At the outset, we offer the reader a small sample of 
definitions which capture the range in the field. 

A criterion-referenced test 1s one that is deliberately constructed so 
as to yield measurements that are directly interpretr.ble in terms of 
specified performance standards (Glaser & N1tko, 1971, p. 653). 

A criterion-referenced test Is used to ascertain an Individual's 
status (referred to as a domain score) with respect to a well-defined 
behavior domain (Popham, 1975, p. 130). 

A pure criterion-referenced test Is one consisting of a sample of 
production tasks drawn from a well-defined population of performances, a 
sample that may be used to estimate the proportion of performances In that 
population at which the student can succeed (Harris I Stewart. 1971. p. 2). 
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The history of norm-referenced achievement testing has been described 
in part by a range of scholars, each operating from a differing frame of 
reference (Nifenecker, 1918; Spearman, 1937, Cronbach and Suppes, 1969; 
Buros, 1977; Levlne, 1976). The particular path of development of 
criterion-referenced testing is less well documented, although partial 
attempts at description have been produced by Mill man (1974), Brennan 
(1974), Popham (1978), Hambelton (1978), and Baker (1980). Under 
contention, for example, 1s when criterion-referenced measurement (CRM) 
began. It seems to have two major sources: curriculum development Inquiry 
and instructional psychology. Its early roots can undoubtedly be traced to 
Rice's assessments (1893), continued with Thorndlke's experiments (1918), 
and Washburne's applications to school objectives (1922). The Impact of 
Ralph Tyler's contribution cannot be underestimated, with Ms widely 
disseminated writing on curriculum development and evaluation (Smith and 
Tyler, 1942; Tyler, 1943; 1950; 1951). There 1s similar evidence, from the 
work of instructional psychologists, of the early development of CRM 
techniques for the assessment of Instruction, for Instance, films produced 
for World War II training (Hovland, Lumsdalne 4 Sheffield, 1949). In these 
early examples, content was sampled from the Instructional universe of 
films, as is recommended currently by CRM specialists. The psychological 
bases of CRM was ^ater exhibited In the experimental analysis of human and 
animal behavior (Skinner, 1958). 

When reviewing the psychological roots of CRM, the source of 
nomenclature associated with CRM can be Identified. For exariple, criterion 
Itself simply meant a t erminal or ending frame in a sequence of programmed 
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instruction, where the response opportunity for the learner was unprompted 
(or without cues supporting the correct answer). Only later were such 
criterion trials aggregated into a criterion test of the sort Glaser 
described. Programmed Instruction absorbed the attention of many 
psychologists concerned with changing student performance, who provided us 
with concepts such as task analys s (Gagne, 1965; 1977), performance level 
(Mager, 1962), and individualized instruction (Holland and Skinner, 1961; 

Lindvall and Cox, 1969). 

CRM was first conceived to be a depenr?nt measure for instructional 
sequences, sequences which were concrete and carefully designed. Thus the 
purpose of CRM was twofold: 1) to provide an operational definition for 
the skills developed by a given sequence, 2) to be used as a mechanism for 
formative evaluation (Scriven, 1967) as a way to improve Instruction. The 
use of test information to revise instruction was a tenet of programmed 
instruction, and was also called developmental testing (Markle, 1967) or 
field trials (Lumsdaine and May, 1965). Of great importance, however, was 
that the test and instructional sequence were intimately connected, which 
made elaborate description of what the test measured unnecessary. 

Early Applications 

Fed by both the programmed instruction movement and the broader 
curriculum development and evaluation concerns of Tyler (1950) and Bloom 
(1956) was the movement in American education relating to behavioral 
objectives. Advocates of such objectives (Mager, 1962; Popham and Baker, 
1968) argued that specification of goals allowed teachers greater 
efficiency in their instructional tasks as well as concrete means for 
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assessing the success of their instruction. Although the movement often 
resulted In enthusiastic overspedflcation, with hundreds of tasks 
identified for a single course, the progressive refinement of the idea 
resulted in fewer objectives (to aggregate discrete objectives Into 
clusters that were more sensible for learning and Instruction). The 
emergence of more general izable classes of behavioral goals and the 
recognition that the evaluat' . of these goals (testing) needed to derive 
from the clear statements leu to the development of specification-oriented 

testing, or CRM. 

From the Tyler tradition, and elaborated by the work of Carroll 
(1963), Bloom (1968), and Keller (1968), teacher-oriented notions of 
m attery learning developed. These models shared an important philosophic 
view, adopted, it appears, from the work of the programmed instruction 
designers: that student success was the shared responsibility of the 
teacher and the learner. Teacher training models were concomitantly 
developed, based on this point of view (Michigan State University, 1968; 
Popham and Baker, 1970; 1973). In addition, the curriculum development 
renewal, spurred by Federal support of regional educational laboratories 
and research and development centers (Title IV, ES£A, 1965), Integrated 
Tylerlan and programmed Instruction traditions (see for example, products 
developed by the Southwest Regional Laboratory 1n California, or the 
Learning Research and Development Center, University of Pittsburgh). These 
instructional systems, whether purely programmed Instruction, teacher- 
mediated, or comprehensive systems, depended for their evaluation on 
quality criterion measures. Thus, the Initial utility of CRM was almost 
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always as a part of an instructional system. The tasks assessed by CRM 
were circumscribed by the goals of the Instructional system. 
The Beginnings of CRM as a Field of Study 

As shown earlier, critical J \1ons of CRM include the notion that 
performance is assessed relative a> a particular task domain and that 
representative samples of tasks from this domain are organized to make a 
test (Glascr and N1tko, 1971). Glaser's work spurred the analysis of CRM 
as a measurement model rather than only as a part of an instructional 
system. 

Early discussions cf CRM, after Glaser christened the fledgling 
approach, struggled 4 .o contra?* CRM from traditional testing theory. In 
their well known and referent article, Poph .1 and Husek (1969) contrasted 
CRM and norm-referenced tests (NRT) cn the basis of test development 
procedures, test Improvement procedures, analysis and interpretation 
routines. NRTs were so named because their reporting procedures required 
that individual scores be transferred to a common scale and characterized 
as ranks 1n a distribution of scores. Thus, a score had meaning only 1n 
comparison to other scores in a particular distribution. Data were 
reported 1n terms of percentile, stanine, or quartile. It became gradually 
clearer to researchers that the norming process not only depended upon the 
selection of appropriate comparison groups of students, but also that it 
significantly influenced the development procedures of the test items 
themselves. The development procedure was bound by the requirement of 
performance variance to permit normal curve interpretation. Thus, early 
distinctions between norm- and criterion-referenced tests we " drawn in 
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terms of what was expected to happen to this variance after Instruction. 
Because norm-referenced tests were developed to provide discriminations 
among Individuals and relatively stable estimates of Individual 
performance, instruction was expected to affect students about equally. 
The shape of a norm-referenced score distribution would not change as a 
function of instruction. Everyone was simply expected to move up a few 
notches (as the phrase grade-equivalent suggests).- The relative rank of a 
student's score in a distribution was not expected to change. In contrast, 
criterion-referenced score distributions should alter dramatically after 
the treatment of related instruction. Before teaching, the pretest 
distribution migh* be homogeneously clustered and low on the scale for 
peculiarly obscure tasks, or for more general areas, randomly distributed; 
following instruction, it was conceivable for the great proportion of 
students to be achieving very high levels of performance, with relatively 
small variance. Before too long, researchers recognized the effect of 
reduced score variability on the utility of extant statistical procedures 
for examining test adequacy. 
The Problem of Identity 

Just as a young child probes the limits of his own identity and seeks 
to separate ar1 distinguish himself from his parent, so did the writers in 
the area of CRM continue to seek to differentiate CRM from norm-referenced 
testing. Streams of articles attempted to descrl^ what CRM was, including 
Popham and Husek (1969), Simon (1969), Lindqulst (1969), Ivens (1970), 
Block (1971), Ebel (1971), Harris and Stewart (1971), Glaser and Nitko 
(1971), Emrlck (1971), Cronbach (1971), Kriewall (1972), and Livingston 
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(1972) Much of these discussions focused on the model underlying CRM. 
There were two basic points of contention. First, the question was raised 
whether the term criterion meant a criterion set of behaviors, or 
essentially a task domain, whether 1t meant rather a standard or 
performance level, such as 70% of the Items correct, or whether It was to 
be used as an external criterion, such as in criterion validity (Brennan, 
1974). A second point of contention was how well specified were the 
domains from which the Items were drawn. Some suggested that a CRM needed 
careful specification of both content and behavioral domains. The 
recognition of different degrees of specification led to analyses which not 
only contrasted norm and criterion-referenced tests, but also attempted to 
distinguish subsets of CRM, such as ob.iectives-based, domain-referenced, 
and ordered sets . (See, for e ample, Denham, 1975; Sanders and Murray, 
1976; Skager, 1975; Harris, Alkin, and Popham, 1973; Glaser and Nitko, 
1971; Millman, 1974; Popham, 1978; Dzulban and Vlckery, 1973; Hambleton, 
Swaminathan, Algina, and Coulson, 1978; Berk, 1980; and Baker and Herman, 
1983). The recency of some of the entries suggests clarity 1s not rampant 
in the field and, 1n fact, which concepts are subsumed by which appears to 
be a matter of personal preference by various writers. 
Conflict 

A good many of these articles and books attempted to distinguish 
between CRM and NRM by casting doubts on the goodness of one or the other 
(see, for example, Perrone, 1975; Haney, 1979; Ebel, 1972). Such doubt, 
were easy to support on either side, for assessments of the quality of 
available commercial achievement tests, both norm referenced, (Hoepfner, 
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1971-1976- Haney, 1978) and criterion referenced (CSE Test Design Project, 
1979) were generally negative. 

From the literature alone, it is difficult to gauge the intellectual 
environment in which these discussions occurred, but in fact, a good deal 
of rancor was generated by contending advocates for norm and criterion- 
referenced testing. Within active memory were rather vitriolic exchanges 
between purveyors of the "upstart" form of assessment, the CRM devotees, 
and those firmly grounded in traditional psychometric theory. Debates were 
held at resesarch associations. National professional groups published 
resolutions in favor of one or another sort of testing, and then sometimes 
switched sides. A joint committee of the American Psychological 
Association, the American Education Research Association, and the National 
Council for Measurement in Education (1974) made an attempt to mediate 
differences (American Psychological Association, 1974). CRM advocates saw 
themselves as student and teacher oriented, interested in testing in the 
name of formative evaluation and the improvement of education. 
Norm-referenced test authorities held fast to the long and scholarly 
psychometric traditions upon which NRT was based. They could point to well 
developed concepts of individual differences, robust parametric analyses to 
assess the quality of their measures, and a thriving industry of users. 

The sum of the criticisms of CRM by this group was that it was largely 
atheoretical nonsense. Should one review some or the early examples of 
CRM, such criticism is clearly appropriate. As will be detailed later, 
test construction in the name of CRM proceeded at a superficial level. 
Items were generated and reviewed under less than rigorous conditions 
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(justified, of course, because the empirical analyses available to improve 
norm-referenced tests could not be directly applied and interpreted for 
CRM). 

Social Context and the NRT-CRM Debate 

One of the great ironies of this period of CRM development, the late 
sixties and early seventies, occurred as a function cf the social reaction 
in American education. Precisely at the time CRM was emerging and 
differentiating itself under the banner of more educationally and 
instructional ly relevant assessment, a strong reaction to technology of any 
sort took place. Both NRT and CRM advocates were tarred by the same brush 
by representatives of the counterculture, activists who rebelled against 
institutionalized testing and its attendant philosophy of logical 
positivism. Thus, CRM and NRT were thrown together as "the enemy", and 
distinctions between models of assessment were overshadowed by the general 
rejection of "irrelevant" and competitive educational activity. These 
reactions, scholars avow, were in part caused by social disruption, the 
limited success cf the Great Society (Aaron, 1980), and evidence of the 
perversion of public political power. 

At the same time, and causing additional conflict in the practical 
world of education, was the increasing public attention and support of 
testing (Atkin, 1980). The evaluation requirements attached to Federal 
categorical aid programs spread the amount of testing throughout the 
nation. The interpretation by the courts of test data, such as reported in 
the Coleman study (1966), the trends toward statewide achievement programs, 
and the development of school leaving examinations as a criterion for high 
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school graduation (Pipho, 1978) raisei the testing stakes. What had 
started as an academic squabble between educational psychologists grew to 
an isrue of considerable proportion In public policy. As the testing Issue 

.a.ne more visible, and Involved life choices of Individuals, so did the 
need to i-ientlfy problems 1n the testing field become more urgent. 
Consumer advocate groups (such as Nader's) attackeu testing Institutions, 
questions regarding test security were raised concomitantly (Haney, 1978), 
teacher organizations presented forceful points of view (NEA, 1979; Ward, 
1980), contention was fed by court cases and legal analyses of tests were 
Issued (McClung, 1978). Another broad Irony 1s that most of these analyses 
of test properties were based on work of psychometrldans, a professional 
group with restively little school experience and almost no Involvement 
with Instructional program. 

Especially noteworthy 1n reviewing tne development of CRM 1s that only 
rarely were the core philosophic distinctions between NRT and CRM clearly 
articulated. Bloom (1968), In his classic article on mastery learning, 
pointed out the difference 1n expectation such a model could make for 
children and outlined some of the benefits of allowing learning time rather 
than student competency level to vary. One clear consequence was the 
sharing of Instructional responsibility by teacher and student. Not yet 
solved, however, are tfr practical difficulties of implementing such an 
idea 1n the face of continued social and financial pressures 1n schools. 
These difficulties Include problems a^oclated with realocatlon of 
resources to students who require more time, the nature of shared 
responsibility In the face of high student absentee rates, and the tendency 
for mastery to be set at lower rather than higher levels (Baker, 1978). 
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Test Design for Criterion Referenced Measurement 
When one Imagines what ought to be In a section called test design, a 
prominent contender 1s how to make a test, that is, the nuts and bolts of 
actual item writing and test assembly. While such activity has rarely been 
reg-ded as at the higher end of the Intellectual continuum, nonetheless 
rules, procedures, and routines for test construction have been developed, 
for use by either the professional test builder or by teachers. In this 
section, some contrasts will be presented between test construction 
activities and test design efforts, the former characteristics of typical 
achievement test development and the latter examples of test development in 
CRM. 

Norm-Referenced Test Developmen t: In Brief 

Certain steps In achievement test construction were developed in 
traditional practice. It should be emphasized that the routines were 
created 1) to assure a broad representation of item and content types; 
2) to avoid gross technical error. The major burden of test development 
for norm-referenced achievement tests (NRT) fell on empirical analyses. 

Typically, in NRT, a general content-behavior matrix ws first 
developed, so that test Items could be generated to tap the full range of 
topics and eligible response modes. Then Items were reviewed to assure 
that they did not Inadvertantly cue the learner to the correct answer, that 
the length and syntax of response options were comparable, and that the 
correct answer was keyed accurately. ,hese Items were also Inspected for 
content quality and screened for obvious technical errors. Most Important 
in test development processes, however, was the use of empirical procedures 
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to determine test quality. Techniques such as Item analysis, reliability 
estimates, and quantitative indicators of validity were created to help the 
test Item selection process. These techniques were based upon parametric 
statistics used by researchers in analyzing experimental data. Such 
techniques depended, as did certain experimental research models, on 
classical notions of science: predictability and control. 

Underlying empirical test refinement practices was a relatively simple 
Idea. A norm-referenced achievement test was to measure a general ability, 
pertinent to an area of knowledge or skill. The underlying "explanatory 
concepts... accounting for test performance" were called constructs 
(Cronbach, 1971). An Individual's performance included chance exposure to 
relevant experience, broadly aggregated, as well as to 1n-school or other 
purposive instructional experience. Constructs, deflnitlonally, required 
more than one measure. Performance on any single test measuring a general 
construct (such as reading ability) was thought to provide a relatively 
stable estimate of an individual's performance when compared to other 
similar Individuals. The role of change (as in learning due to 
instructional exposure) was noticeably unclear. As such achievement 
measures were to assess important dimensions formulated as constructs, the 
argument ran, then they shouM not be reactive to relatively small 
variations -n the learner's total experience, for instance, whether or not 
a child received a particular one month reading comprehension program. 
Such a model was almost universally accepted and maintains strong and 
eloquent supporters (see, for example, Ebel and Anastasi, 1n Schrader, 
1980). They describe a view of achievement as a developed ability, with 
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the other end of a continuun anchored by aptitude (the capacity or 
predisposition, without the relevant experiences). This notion of 
achievement was supported by statistical analysts who conceived of testing 
in terms of prediction. Changes in test score from occasion to occasion 
were formulated as unreliability o. error (see, for example, Harris, 1962) 
by such methodologists. 

Certainly, no one worries much about models underlying test 
construction or any othe; human endeavor when certain conditions hold: 
(1) performance looks good; (2) significant decisions do not hinge on the 
model's products; and (3) a body of prestigious support is available for 
the practice. Such was the comfortable status of norm-referenced 
achievement testing for many years. Measures now show a less than rosy 
view of student achievement, and explanations for declines have not been 
satisfactory (Wirtz, 1977). Decisions about admission to professional 
schools, coveted undergraduate institutions, and even the award of the high 
school diploma increasingly depend upon test performance. Obviously 
important, perhaps, is the lack of scholarly consensus on the quality and 
utility of achievement measures. Because these Issues focus attention on 
the effectiveness of schools, a different philosophy about education has 
developed vocal, 1f not always coherent, support. That view Is also 
simple: that schools exist to produce change, in other words, specific 
learning. In this view, change 1s not regarded as score unreliability, but 
1s itself the most desired product of education. One should note the level 
on which discussion of this Issue has occurred. Secretary Joseph Califano, 
then head of the Department of Health, Education and Welfare, made a public 
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statement where he avowed that the federal government wished to reduce the 
predictability of performance based on socioeconomic or race 
classifications (1978). Since relationships in status on these demographic 
variables and standardized test performance run very high (between .60 and 
.80) depending upon the reliability of the test, and student performance on 
similar tests correlate, over time, at .80 or higher (Bloom, 1980), one may 
Infer that this statement challenges the test development community to 
build measures able tc detect effects of educational practices within the 
school's control. In contrast to earlier formulations, change 1s to be 
valued over predictability. This perspective shift has great Implications 
for test construction. Procedures used to develop measures of traits 
thought to be essentially stable over time are not the same ones that 
should be used to create change-responsive outcome measures (O'Nell and 
Richardson, 1977). 
Specifications of Tasks 

CRM developed, 1t was earlier noted, cut of two traditions, each 
actively promoting change: instructional psychology and curriculum 
development. Both of these sources, although from different governing 
frameworks, hit upon the practice of specifying objectives or goals for 
change. The practices In CRM development grow from the answers to various 
questions related to this specification or description: What 1s 
specified? At what level of detail? Where do the specifications come 
from? 

In the earliest days, specification of tasks for assessment were 
thought to flow very nicely from a clear statement of an Instructional 
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objective (Hager, 1962). Although these objectives could be developed to 
cover course-level material, they were usually created for shorter units of 
instruction. The belief was evident that, once figuring out how to state 
an objective clearly, test development woulc be a cinch. In rules designed 
to help in the assessment of educational programs, Popham (1n Baker and 
Schutz, 1968) suggested that the critical measurement Issue was the 
classification of forms of stimuli and responses. As an early advocate of 
diverse forms of measurement, Popham classified assessment tasks Into four 
cells: (a) student behavior could be either process (throwing a ball) or 
product (test paper); (b) ellcltatlon conditions could be either formal 
(school) or natural (out-of-school or surreptitious). Additional writing 
around this time focused on how specific the specification needed to be for 
the assessment ("to take a test" was a negative example, considered much 
too vague). Also of Interest were conditions under which the test was to 
be taken (time limits, extra materials) and ways of establishing desired 
performance standards (such as 75% correct). While Tyler and others since 
had noted that an objective consisted of both behavior and content, a good 
deal of early attention In objectives-referenced measurement was devoted to 
specifying behavioral requirements and very little In developing the 
content oarameters. Good Items were thought to match the behavioral 
statement 1n the objective. 
The Problem of Content 

In the absence of routines for specifying Che what (content) of 
testing In favor of the how (test behavior), two rather different modes of 
practice developed. Test Items were selected or rejected on the match 
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between the objective statement and nuances of the test taker's behavior 
(was the student directed to cross out a letter when the objective calltd 
for a machine scored blacked in response?). In one mode, content was left 
to vary freely without any specification ("important mathematics concepts" 
or "American novels"). In the other, each particular content unit was 
specified ("In the play Othello, identify..."). The trade-offs appeared 
clear: in the first case, the task was cast in a generalizable form, for 
almost any particular content would be eligible for int usion in the test. 
In the second, parti cularizati on of content allowed for highly targeted 
instruction and congruent testing, but forsook general 1zabil1ty. 
Discussions of the merits of these trade-offs, general izablHty vs. 
specific content, were heid in workshops and training sessions of the 
American Educational Research Association during years from 1967 to 1973. 
However, real confrontation with the content of tests, that 1s, the subject 
matter areas to be assessed, was generally limited. Although there were 
analyses of new curricula, new math, the process-oriented new sciences, the 
new linguistics, such were not specifically analyzed for their utility 1n 
developing performance-oriented Instruments. Content people were generally 
too ''soft" for the hard edged requirements of behaviorism, and remarkably 
few content specialists were Interested 1n testing specifically. During 
the m1d-s1xt1es, an Impetus for a new view of content In objectives-based 
testing was needed. 
Domain-Referenced Achievement Testing 

The work of Osburn (1968) and Mvely (et al., 1968) provided that 
impetus. Using a model developed from set theory, Hively described the 
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identification of a universe of content and behavior, a domain. Hiveiy 
demonstrated that broad classes of performance could be assessed by using 
algorithmic rules co generate items. This domain could then be 
theoretically sampled to yield representative instances of test items. 
Performance on the sample would allow the estimation of performance for the 
larger content/behavior domain. Hiveiy, in refinements with colleagues 
(ig73, 1974) demonstrated how a technology for domain-referenced test (DRT) 
generation could be developed. He suggested the use of an item form, or 
shell, that included basic behavioral requirements. Into this shell could 
be inserted replacement content instances, substituted from the 
"universe". A simple example of an item form is the addition problem: 

x + y - 

where x is any two digit number and y is any one digit number. While the 
item shell might be changed to: 

x 

the content parameters would be identical. Two digit and single digit 
numbers were to be added. Any members of that set In the specified 
combination might actually show up as a test item. 

Hiveiy' s suggestions had great impact for a number of reasons. First, 
as described earlier, there was dissatisfaction with extant test 
development processes In the field. While there was recognition that 
available empirical procedures were Inappropriate to apply to new outcome 
measures, no alternative procedure had been agreed upon to produce quality 
test Items. Hively's work probably also Indirectly capitalized on the 
widespread knowledge of Bloom, Krathwohl, and colleagues' (1956; 1964) 
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efforts at taxonomic organizations of educationa. objectives. The term 
dpjaln used in these works was understandable to all. An additional 
explanation for the success of Hively's ideas was his development and 
demonstration of domain-referenced achievement testing in concrete form. 
He provided a real example to researchers in the field, an example couched 
in a theoretical context but which had practical implications. He had 
actually created test items using such procedures. 

Forms of Items Forms 

hively's rules for the creation of items included the specification of 
the formal of the item, the rules for generating the stem, the response 
alternatives, and the directions. When fully explicated, Ms Item form 
directions appeared detailed and formidable. Such detail was clearly 
required in order to develop unambiguous item domains. Yet his procedures, 
because of their sophistication, seemed designed principally for use by a 
team of item writers. Baker's adaptation, reported in Hively's book 
(1974), focused on specifications as they might be modified for teachers 
and others familiar with behavioral objectives. The elements of a domain 
specification included a statement of the objective, the content limits, 
the wrong-answer population (for multiple choice tests) or response 
criteria (for production tasks), the item format, the directions, and a 
sample item. Popham (1975) further modified domain specifications to what 
he termed an amplified objective . In his scheme, stimulus attributes and 
response attributes were to be specified; however, distinctions between the 
behavioral and content requirements of the item were not made. The Popham 
and the Baker adaptations represent less rigor than the Hively approach, 



ERIC 131 



24 



but were justified 1n terms of likely comprehenslblli ty to teachers and 
Instructional designers. At the outset, these approaches were applied to 
single domains and the problems of creating tests across a number of 
related domains was not addressed. 

Hlvely's work was particularly Important because of Its connection 
with Instruction. Unlike the curriculum development people, who saw 
specification of objectives and measures as one of the first steps 1n the 
process, Hlvely had directly referenced his efforts to extant Instruction. 
He used content generated by lesson writers as the primary source for the 
creation of his Item domains. Similar to the way 1n which programmed 
instruction linked Its criterion-frames to Instruction, so Hlvely's Item 
forms were linked to the concepts 1n actual lessons. Although his work was 
extended by Popham, Baker, and others to the objectives-instruction- 
assessment sequence, his Ideas remained firmly grounded 1n Instruction. 
Domain-referenced testing (DRT) Immediately formed a new category of 
criterion referenced measurement, and writers described applications in 
teacher training, program evaluation, and accountability (see, for example, 
Hlvely, 1974; Harris, Alkln, and Popham, 1974). 

DRT generated fodder for Intellectual rumination lasting well Into the 
most recent period. Questions were raised, and almost endlessly discussed, 
by Popham (1978), Mlllman (1974), Hambleton (1978), Baker (1978), Brennan 
(1974), Harris (1980), Haladyna and Roid (1978). N1tko (1974), and Anderson 
(1972). Numerous problems 1n DRT were Identified and lists of unresolved 
problems published 1n 1974 appear to continue in that status. 
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Problems of Domain-Refe renced Testing 

Among so« of the early problems associated with DRT was t» attempt 
to deal with cohteht parameters outside the field of mathematics and 
science. Although It was very clear how one might go about generating a 
set of parameters or generation rules for computational questions, doing so 
1n the liberal arts appeared to be , mess, process. Hlvely's procedure was 
based upon an algorithmic approach to content selection. Tbus it was 
especially applicable to content areas that had well-defined structural 
relationsntps, sucb as an early example of DRT in a linguistically oriented 
reading program (Baker, 1968 In Ml example, a specific set of rules 
governing content, such as syntactic and spelling rules, allowed for the 
explication of a universe of content and the compilation of tests that 

sampled the defined universe. 

The attempt to apply DRT to otber subject-matter areas were many, and 
included social studies, writing, English literature, the bealtb sciences, 
and reading comprehension. A major fact soon became evident: few subject 
natter areas bad sufficiently well-defined structures to permit the use of 
algorithmic approaches to content generation (Landa. 1974). In the absence 
of sufficient clarity 1n subject matter fields, would-be users of DRT fell 
back on an alternative process. Their choice was to define the parameters 
of content operationally themselves, without reference to any subject 
matter analyses. They would decide, for example, that four causes of 
economic decline existed, 11st and define such causes, and develop examples 
of each. A DRT could then be created by selecting an appropriate range of 
examples. This method was clearly vulnerable to cha^s of both 
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arbitrariness and curriculum control. Defenders of this strategy pointed 
to the void in current practice and suggested that this technique was 
preferable. As a coincidence, Gagne (1977), In an audiotape developed for 
AERA, discussed two forms of concept learning. The first type, concrete, 
were those derived from perception. The second category of concepts were 
those he called defined concepts, where the instructional designer (or test 
writer) would explicate the dimensions of a concept and the learner would 
discriminate examples or generate instances based on these defined or 
agreed upon limits. The use of such defined concepts supported the DRT 
content specifications. A large and unresolved Issue remained: who was to 
decide on the arbitrary features of a defined concept. No satisfactory and 
practical answer.* have been suggested, from the measurement community 
beyond the usual discussion of constituencies and judgment by reasonable 
persons. The advances in cognitive science, however, presage Improvement 
1n specifications. Both cognitive skills and precise content 
representation may contribute to resolving this Issue (Curtis & Glaser, 

1983; Baker, 1985). 

A second major problem was what to do 1n cases in which the subject 
matter itself defied algorithmic definition, even an arbitrary one, in a 
case such as literature. While 1t 1s conceivably possible to specify 
arbitrary rules for generating examples of lyric poetry, the exercise seems 
relatively futile because of the variation of examples within that literary 
genre. Taking a cue from Hlvely, some DRT writers Identified domains not 
by generation rules (for all possible instances) but by enumeration of a 
limited set (for instance, poems 1-9 found In Smith's anthology). Such a 
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tactic reduced the power of DRT to claim estimation of a total domain (such 
as poetry), reduced the likelihood of generalization (that perhaps 
performance levels would be similar from poem to poem), but preserved the 
"fairness" with which Items might be sampled by circumscribing the set of 
content to that contained 1n the particular anthology. Thus, at least, 
students and teachers and test writer would know what content was fair game 
for testing. 

Another fall-back tactic for content specification was to define by 
illustration and axiom a set of content. Mvely -provided the example of 
the frontpage of The New York Times as a content set for assessing reading 
comprehension. Clearly the explication of generation rules or algorithms 
for content such as The Times 1s beyond both the funds and attention spans 
of researchers. In another example, the operational definition of a clear 
sentence , Including forms of reference, semantics, and soon, similarly 
over-complicates a domain more Intellectually accessible by example. As 
provided 1n any number of style handbooks, clear sentences can be clearly 
contrasted with unclear writing. The rules are more efficiently perceived 
in the examples themselves, rather than exhaustively written. Again, this 
form of specification, while short of the purity of Item generation rules, 
clearly communicates to teacher and learner what 1s to be tested and what 
should be learned. 

The problem of the completeness of content domain specification can be 
recast as a problem 1n automation. How fully automated should DRT's be? 
The extent to which test Item writing can be fully automated 1s presently 
unknown but approximations using domain specifications or syntactic rules 
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have been attempted. Bormuth (1970) provided essentially linguistic 
transformations to permit the generation of test items. In a series of 
studies to assess the automaticity of item writing, Rom and Haladyna 
(1978) were surprised that item writing "subjectivity" was not removed by 
the provision of rules to two item writers. In another study using prose 
passages, Roid, Haladyna, and Shaughnessy (1979) found some algorithmic 
practices controlled item writing production. The study supported the 
importance of linguistic analyses of items in addition to other 
specification matching routines. This study was also limited, however, by 
the use of only a few (four) item writers. Undaunted, they continued 
(Roid, Haladyna, and Shaughnessy, 1980) with six Item writers directed to 
use linguistic vs. subjective (match with an objective) strategies. 
Although lengthy analyses are provided, the item by Item writer interaction 
suggests that Item writer behaviors were not sufficiently effected. The 
authors posit the need for further trials with more empirical tryouts. 
However, tryouts under conditions of good, medium, or rotten instruction 
would likely affect the resulting data set. Baker and Aschbacher (1977) 
achieved considerable success in controlling item production through the 
use of rules. The automation problem has not been discussed in most 
research 1n this area. The use of the computer to automate Item writing 
routines has been less well -developed to date than one might hope, with 
only relatively simple content substitutions used. Hlllman and Outlaw 
(1977) conducted a project in this area and Finn (1978) reported on 
multiple-choice Item generation. Hsu and Carlson (1973) earlier used the 
PDP-10 system, and other automated experiments Involved efforts by Olympia 
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(1975) and Fremer and Anastasio (1969). This work needs to be linked and 
made more relevant to the content parameters of domains. Perhaps 
availability of better natural language processing options would Improve 
computer utll'zatlon In this Important area (see Frase, 1980; Freedle, 
1985). 

Ne w App~oaches to Content Sped flea tic . 

While computer technology has long been employed to v i and to 
administer tests (Dunn, Lushene * O'Nell. 1972; Hedl, O'Nell * Hansen, 
1973), Its exploration may have some utility in the content specification 
problem of domain reference achievement testing. Specifically, the 
development of expert systems provide an opportunity for specific knowledge 
domains to be Identified, structured and Incorporated Into computer 
software. Basically, these approaches focus on the problem of representing 
expert knowledge and its lationshlps In algorithms that the computer can 
use {Buchanan, 1981). Mc .elling knowledge via expert systems have, by and 
large, focused on relatively narrow knowledge domains, such as subtraction 
(Brown ft Burton, 1978), but efforts have been made to attack npre complex 
areas, such as computer programming (John ft Soloway, 1985), infectious 
diseases (Clancy, 1982), story generation (Dehn, 1981? and understanding 
narrative (Dyer, 198?. and Fre<ier1cksen S Warre... 1985). Research 1s also 
underway to develop procedures for less well refined areas, so called fuzzy 
content (Spiro, 198** where content does not fall Into mutually exclusive 
categories. The techniques used to represent knowledge developed for AI 
expert systems could be used In the vexing problem of assuring full content 
represen cation on tests. 
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quality Control 

Another nagging question about DRT 1s how one knows an Item is a good 
instance of the set. Most writers suggest some judgment -scheme, usually 
matching the.item realistically against characteristics explicated In the 
specifications. Research on this problem has demonstrated that raters may 
make their discriminations on superficial Item features; for example, does 
the number of response alternatives In the item match the specifications? 
rather than on the more difficult Issues of cognitive complexity or content 
appropriateness. Some research has been conducted relating to the need to 
provide guidelines for such judgments (Polin and Baker, 1979). 

Using defined concepts and operating from an instructional 
perspective, rules and routines for matching Instances with classes have 
been developed by Markle and Tlemann (1974), Tiemann, Krocker, and Markle 
(1977) and Tlemann and Markle (1978a,b). Merrill and Tennyson (1977) have 
also provided excellent analyses and examples of processes needed to match 
examples of concepts to specifications or concept definitions. Because 
this work takes place in the context of Instructional rather than test 
design, these authors have received less than their due recognition for 
contribution in the testing field. 

Of the research conducted on providing guidelines for judgment in a 
test design context, Hambleton (1980), Haladyna and Ro1d (1977), Baker and 
Quellmalz (1977), Doctorow (1978), and Polin and Baker (1979) have made 
contributions. Set theory, or more particularly the concept of fuzz y / sets , 
has hen applied 1n this research tc estimate the degree of congruity 
between an Item and its specification. This research demonstrates the 
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futility of using obvious and superficial Indicators (such as the number 
foils 1n the specifications); and factors such as level of cognitive 
complexity and related linguistic features were highlighted as needing more 
study. A number of writers have reported training efforts undertaken to 
teach specification - Item matching (Merrill. 1979; Tlemann and Markle, 
1978; Hambleton and Simon, 1980). Baker. Pol1n, and Burry (1980) have 
developed training materials designed to teach the rudiments of DRT 
judgment to teachers and to graduate students. Such training seems to be 
required before Individuals can match test Items with their 
specifications. Secolsky (1980) makes the argument that students must be 
able to match relevant Items with their generation specifications (I.e., to 
label concepts, to demonstrate that the Items cohere). This rather 
demanding requirement might be acceptable 1f students were first trained 
specially 1n identifying the critical attributes 1n DR Items. In the 
absence of such training on relevant dimensions, students might group Items 
under true, covarying but Instructional ly Irrelevant features (such as 
sentences starting with the letter T). In the development of the review 
process described earlier Investigated by Polln and Baker (1979), the 
critical issue was training Item classifiers on instructionally relevant 
Item features. 

The foregoing problems that deal with the match by Inspection of 
specifications and Items represent what Bormuth (1S70) calls problems of 
item-writing theory. His second category deals with Item-response theory, 
or more accurately empirical Indices used to substantiate the existence of 
a domain. Ml lima n (1974) also attempted to distinguish between problems of 
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Item selection which were judgmental and those for which empirical data 
were necessary. Popham (1978) also distinguished between descriptive 
validity (that 1s, does the Item fit its specifications and are those 
specifications clear?) and functional validity (does performance classify 
the student as anticipated?). Early Interpretations of the DRT process 
included high expectations of Item homogeneity, as discussed by Nitko 
(1973). The Idea was that Item difficulties and variances for Items 
produced by DRT procedures should be similar. Items were expected to 
cluster together (Baker, 1971; Macready * Merwln, 1973; Stenner & Vaster, 
1971). Cronbach (1972) discussed procedures where individual Item writers 
would be able to produce Items which resulted In similar empirical 
characteristics. Although this demand for homogeneity has diminished in 
the light of actual data sets, one nay stm be troubled by the idea that 
item performance, particularly one developed by DRT procedures, was 
assessed in the absence of clear documentation of the instructional 
conditions ^receding its use. A similar Issue may by looming for the 
advocates of new empirical procedures thought to obviate the requirement 
for meticulous matching of - pec^ flcatlons with items. The Rasch model 
(Wright, 1967) has been put forth and scooped up by users of CRM as an 
empirical solution to the issue of Item quality. What is still unclear, 
however, 1s the extent to wMch this model, and .i fact other latent-trait 
(Boch, Mls^evy, & Woodson, 1982; Boch, Gibbons, & Murchl, 1985) models are 
robust in the face of highly targeted Instructional interventions. 
Research by Ro1d and Haladyna (1980), albeit exploratory, does not lead one 
to expect good news. Somehow empirical analyses, combined with judgment of 
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specification to Item matches, conducted under known instructional 
interventions, will be necessary before we can uncritically adopt solutions 
such as the Rash model proposes. 

Matching Items to specifications or the generation of Item sets 
according to specifications 1s based on a pigeon-hole view of the 
relationship of given Items to a domain. Each Item would be sorted as 1t 
fits according to the exhibition or absence of N features explicated In the 
domain specification (Choppln, 1980). It 1s altogether possible that 
limitations of Item writers, subject-matter structure, and technology will 
conspire to promote alternative, perhaps supplementary models to DRT. One 
such area of analysis Involves the linguistic features of test Items, 
beyond the readability Indices presently computed. A similar technique 
area once again Hpe for exploration 1s the area of facet analysis and 
concept mapping (see Engle & Martaza, 1976; Gutman, 1969; Harris, 1976; 
Beck, 1978). The Improved natural language processing capacity of 
computers may «lso enrich our DRT technology. One principal Incentive for 
such work may be the need for procedures for the development of access and 
retrieval routines for computerized Item banks. Such techniques could 
easily Influence Item development and review processes and result 1n 
significant Improvement. 

The foregoing discussion pertains principally to the technology of 
comparing sets of generated Items with their parent specifications. Only 
oblique discussion has hinted that the content and behavioral requirements 
themselves might requirement review. Along what dimensions might 
specifications be judged? In much the same mode that goals and objectives 
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were to be judged by relevant constituencies, so too might domain 
specifications be reviewed for relevance and importance in school 
learning. Some critical questions still need research before we could even 
begin to open the review process to less technical participants. 

For example, how big is a domain? The answer was at first thought to 
depend upon empirical data (to wit, a domain has items that cohere), but as 
strict expectations for item homogeneity faded, so have guidelines for the 
restrictlveness of domains. How much complexity in a domain? Are 
homogeneous response modes required? Does a domain include the task to be 
tested as well as relevant sub-tasks 1n an identified skill hierarchy? Do 
such subtasks need enumeration or do they also require verification 
empirically? How are domains organized with respect to one another? In 
parallel? itent area? In more than one way? How are task 

requirements best determined? As pointed out, for the most part 
specifications have grown from the analysis of content areas and rather 
gross behavioral requirements. In some cases, instruction Itself has 
generated the parameters. What should be the relationship of Instructional 
analyses to domain design? 
Integration of Testing and Instruction 

The relationship of domain specification to instruction is an area 
which might profitably be addressed. Certain models start with instruction 
or content (see Hlvely, et al., 1973) and reference the domain to that 
set. Others start with the test specifications, and then develop 
Instructionally relevant learning opportunities (see Rankin, 1979). Thus 
from given domains, test specifications, item pools, and Instructional 
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practice exercises are generated. This system does not completely specify 
all Instruction but 1t 1s designed to Integrate some aspects of domain 
design with testing and Instructional functions. In mastery learning 
(Bloom 1969; Block, 1971), a natural oscillation between Instruction and 
testing occurs. 

Researchers are presently at work attempting to find ways to connect 
Instruction and testing at deeper levels than 1n the past. Rather than 
developing tests to reference extant Instruction (see the Proficiency 
Verification System, SWRL) or to map extant tests on Instructional texts 
(Floden, et al., 1980; Porter, 1980; Montague, Ellis, and Wulfeck, 1983), 
ways to unify the design of test and Instruction should be explored. 
Initial development of this sort has taken place with the creation of 
Project TORQUE (Schwartz and Garet, 1982), a math program where exercises 
serve almost Indistinguishable functions of teaching and testing. The 
cognitive specifications for such a set of activities probably needs 
additional refinement. Frase (1980) has worked on the Integration of 
testing and Instructional domains using computerized language projects, and 
the research in writing assessment (Baker, 1982; Baker, Quellmalz and 
Enrlght, 1982; Purves, et ai., 1980; Quellmalz, 1980) has potential for a 
similar sort of unification. Such a merger of Instruction and testing will 
not come about easily. For one thing, 1t violates our traditional patterns 
of thought. Brennan (1974) expresses little patience with those who 
continually blur the distinctions between testing and Instruction and 
Impede, he believes, serious progress 1n either. On the other hand, a 
scholar as prestigious and traditionally grounded as Harris (1980) has seen 
the need to Integrate testing and Instruction complexes. 
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Most writers on Instruction and testing have, 1n recent years, seen 
tjsts leading Instruction, as In "teaching to the test". Mastery learning 
made a great contribution towards the Integration of Instruction and 
testing 1n two ways. First, the Intervals between Instruction and tests 
were reduced and made more frequent. Second, they were Individually 
tailored for Individuals (Rudner, 1978) or groups. Adaptive testing, using 
the computer to administer tailored Items 1s a current example of this 
approach. Thus, the pattern was changed from formal and extended periods 
for testing and Instruction (courses with only one mid-term examination and 
one final examination) to more flexible and naturally occurring events. 
But 1n the hearts and minds of many, Instruction 1s still the treatment or 
Intervention and testing 1s still the dependent measure . 

For an analogous example, recall some of the early processes 1n the 
attempt to teach young cnlldren to read. An Important and persistently 
difficult skill was the blending of Initial consonants and phonograms, so 
that when a child was presented with the elements T and AN, he or she could 
pronounce TAN. For some reason, Instruction focused on reducing the 
Interval between the pronunciation of elements. By shaping the child's 
behavior so that the time between the pronunciation of T and AN was very 
short, the child wou!<* .„e, 1t was thought, to understand the process of 
blending. In fact, no such Insight typically occurred. Children showed 
remarkable resiliency and ability to keep the two elements separate, even 
when the time between them was essentially eliminated. 

Children did learn to blend easily, however, when the focus was not on 
reducing the time Interval, but in changing the framework 1n which the 
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blending Instruction took place. In early experiments (Baker, 1968), 
children were taught to first understand the unified outcome that was 
desired, that the units had meaning, and blending was a process similar to 
saying SAND - BOX. When presented with T + AN no hesitations occurred and 
blending skill became well developed. Similarly, a new dimension must be 
found to underlie both testing and Instruction so that these functions lose 
their uniqueness. Of great promise 1s the work in cognitive psychology, 
which, 1f united with theories of content structure and language, could 
allow the generation of experiences useful to develop and assess, In a 
piece, the desire outcomes of schooling. An excellent analysis of the 
future has been described by Curtis & Glaser (1983). 
Narrow Definition of Testing 

As we discussed, most Individuals writing 1n the field assume a test 
1s a paper-pencil vehicle, usually 1n multiple-choice format. They also 
seem to assume 1) that the test has one correct answer and that other 
alternatives are no more than "foils" to the right answer; 2) that the test 
1s kept separate from Instructional activities; and 3) that the present 
practice 1s probably most efficient. 

There 1s only occasional mention of "performance testing", and a few 
writers grope to find words to distinguish other than multiple-choice 
testing. They use words like appraisal, evaluation, assessment, their 
Roget's litany, to avoid the constrained "test" connotation. In reflecting 
on this review, the reader would be wise, we believe, to make the effort to 
break out of a confined view of testing. The research should be judged as 
1t could or might be expanded to generalize to formats of the sort listed 
1n Table 1. 
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Table 16.1 
Test Format Options 



Format Examples 



1. oral language Formal speeches, conver- 

sational facility 

2. written composition essay examinations, 

expository analyses, 
description, poems 

3. physical activity diving, tennis stroke 

4. creative production art, carpentry 

5. technical exhibition piano recital 



146 



39 



Evaluating Instructional Technology 
One of the most useful options 1n considering outcomes of 
computer-based Instructional Interventions 1s to use the technology of 
delivery as a means of collecting Information related to student outcomes. 
Not only can the computer deliver tests that are embedded in Instruction 
but It can also tabulate Indicators of other Instructional outcomes. For 
example, in the evaluation of a set of computer-based Instruction, the 
latencies of student responses, the numbers of options they selected, the 
frequency with which they selected harder problems can be Incorporated as 
an additional outcome measure of program effectiveness. In some sense, 
these Indicators Involve using processes as outcomes. The student 1s 
encouraged not only to Improve his level of attainment but his fluency and 
exploratory behavior as well. Other automatically recorded Information can 
provide Indices of student attitudes - for Instance, persistence and 
attention. 

It 1s true that scholars working In the measurement area are moving 
toward a fuller concern with the understanding of student learning 
processes leading to particular levels of attainment. For example, L1nn 
(1985) describes a measurement approach that tracks metacognltlve processes 
learners employ as they encounter new reading requirements. Furthermore, 
Shavelson & Salomon (1985), undertake a study of the relationship of the 
symbol system In which the test 1s conveyed and the cognitive processes 
students use to develop their responses. 

The availability of new computer technology for assisting 1n 
assessment problems has both positive and negative sides. On the one hand, 
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It can encourage the Integration of assessment Into the Instructional 
context, so that It 1s more representative, less ceremonial, and less 
artificial than tests of the past. On the other hand, our analysis of what 
has been happening to testing as implemented In new technology 1s 
relatively negative. Short-answer and multiple-choice formats abound, and 
as a result, the performance tested 1s at the lowest common denominator 
possible. Tests, however, only mirror the approach taken toward 
instruction. When tests are molecular and discrete rather than integrated 
and comprehensible, one can make inferences about the quality of thought 
behind the instructional development effort even before seeing the data. 
We expect to see In future assessment, expansion and integration: where a 
common database can be explored to make Inferences about performance, 
levels of attainment, relationships to Individual differences, cognitive 
processes, and attitude development. Such an integrated database approach 
is possible now. However, as long as assessment continues to be regarded 
as the stepchild of instruction, a necessary evil for reporting 
requirements, rather than an integral instrument In the design of 
instruction and the teaching of students, few developers take the risk. 
Integrating Assessment into the Evaluation of New Technology 

While the foregoing sections have focused on assessment and the 
measurement Ideas that underlie It, It 1s Important to place concern for 
outcome measurement In context. What else needs to be Included In the 
assessment of Instructional technology that 1s especially relevant to the 
technological character of the innovation? In other words, what else needs 
to be addressed beyond measures useful for the assessment of non-technology 
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based instruction? Let us turn, for the conclusion of this chapter to the 
issues related specifically to evaluating technology. Our assumption 1s 
that the best ideas posited for the measurement of instructional outcomes 
will be necessary but not sufficient for this evaluation task. 

Assessment, and the evaluation processes which support 1t, is 
represented to be a productive mechanism for the Improvement of educational 
systems and products. And there 1s hard evidence of the utility of 
evaluation in actually Improving technology-based products and efforts 1n 
instructional development (Baker, 1972; Rosen, 1968). Assessment is known 
as well to contain a strong negative potential. Evaluation can Identify 
weaknesses 1n such a way as to inhibit exploratory behavior and risk taking 
on the part of researchers and developers. Playing 1t safe may be seen to 
be the winning strategy. Evidence of evaluation utilization studies 
suggests that when the focus of the assessment is classification or 
accountability (good vs. bad; useful vs. wasteful), the openness of R*D 
project personnel to evaluation processes is Inhibited. Formative 
evaluation, on the other hand, is evaluation whose specific function 1s to 
identify strengths and weaknesses for the purpose of improving the product 
or system under development (Baker, 1974; Baker & Alkin, 1973; S.M. Markle, 
1967; Baker I Soloutos, 1974). The trick, of course, is 1n determining 
what should be studied, in what context the evaluation should take place, 
when evaluation processes are most useful, and 1n skilled hypothesis 
generation about what Improvement options logically and feasibly may be 
implemented. In addition, the Identification of weaknesses (no matter how 
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benign the Intentions of the evaluation may be) creates a documentary trail 
that might be misused by project managers or funding agency monitors. 

These Issues take on special dimensions when the evaluation addresses 
the effectiveness of new technology. All technology development of 
necessity focuses on the Initial problem of system operation: can the 
envisioned delivery system work at all, as opposed to the refinement of 
what the system's merits may be or what effects might be planned or 
Imagined. Outcome assessment 1s often a deferred goal. When dealing with 
emerging technology, the boundaries between technology development and 
science become especially blurred. The creation of technology may be a 
pleasant side-effect for the creator, whose perception of the main task may 
be knowledge production, rather than Instructional effectiveness. 
Intellectual exploration 1s a premium for new technology development, and 
assessment processes can be seen to Inhibit or be Irrelevant to Invention. 

Recent writing 1n the field of evaluation planning has emphasized a 
stakeholder perspective 1n evaluation Implementation. Simply put, this 
means that Interested parties must have an opportunity to understand and to 
shape the nature of the evaluation questions and methods so that they will 
be more Invested in the process and more apt to use any results generated 
(Bryk, 1983). 

With this discussion as context, a special model of evaluation can be 
designed to be adapted especially to the problem of new technologies. 
Briefly, we will detail the features of this model, as applied to new 
technology, a particularly difficult area characterized by weak boundary 
conditions between research and application goals. 
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A Model to Assess Technology 

The model underlying the formative evaluation of technology is 
composed of a minimum set of pieces. They Include the goals or 
specifications, the Intervention, the context of use, the Information base, 
and feedba H loops. Figure _ displays this model linearly, but 1t in 
fac - Ov." v e arrayed 1n a circle or three dlmenslonally. Points of entry 
to the model could vary depending upon th«» designer's committment to 
prespedflcatlon of outcomes, for Instance. Or the extenslveness of 
alternatives could differ, with some designers Interested in contrasting 
alternative instructional treatments and p^ers interested in a broad array 
of outcomes, Including attitudlnal and social goals beyond those detailed 
in the system specifications. 

Figure 

Model to Assess Technology 
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Desired Information Features and Functions 
for a Model to Assess New Technologies 

Below we provide a 11st of four desired attributes for a model to 
assess new technology. These characteristics respond to > ticular 
attributes of technology development. In brief, these Include weak 
boundary conditions between research and application goals of the 
developers; levels of risk 1.1 technology development; and the constant 
pressure to develop and sustain management support and necessary resources 
to complete the tasks of Interest. 

The information must provide an enhanced documentary base for the 
processes of new technology development . A characteristic of new 
technology is lack of documentation describing the process leading to the 
development of the system or product. The purpose of a strong documentary 
base is to provide the trace of developmental processes so that the field 
can Improve overall. Aggregating across a series of case histories of 
projects can allow the inference about productive strategies to be made. 
In addition, a good documentary base can Inform about dead-ends in 
substance as well as 1n developmental processes. Since most R&D reporting 
1s based upon positive findings, it 1^ difficult to avoid useless but 
unreported paths. 

This lack of documentation e<1sts for a variety of reasons. First, 
the proces. of early design cf techno' : y is complex, iterative, and 
non-linear. All of us are familiar with documents of development which 
retrospectively rationalize and make "neat" processes that are chaotic, or 
at bes., hard to track. Furthermore, the metacognitive awareness required 
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of designers to document their own processes while at the same time working 
on problems of Interest presents an almost Insurmountable attention burden, 
even if there w»re predisposition on the part of the research and 
development personnel to do so. Solving the problems at hand appears to be 
more important. Contributing to an abstraction such as R&D processes 
attracts less compelling energy, despite the Intellectual apprehension that 
the field overall can be Improved by a "lessons learned" perspective. 
Another Inhibition 1s the precedence of proprietary knowledge, well known 
1n the private sector, but of potentially Increasing Import in a public R&D 
environment characterized by competitive procurement policies. 

In an attempt to meet this overall goal 1n instructional technology, 
some case histories were prepared 20 years ago (see D. Markle, 1967) and an 
historian was even on the payroll of another large R&D facility. But 
these persons can be as pestering and diverting as media reporters, trying 
to get the Idea of what's going on without true understanding of the 
processes Involved. In new technology development, the problem is 
obviously exacerbated. 

Fully participating formative evaluators provide another model, 
however, 1f they are linked early on 1n the development process, and 1f the 
R&D management and staff understand the Intent is to assist as well as to 

document process. 

The information must use state-of-the-art evaluation methodology, 
including both quantitative and qualitative appr oaches to measurement. One 
of the reasons evaluation processes have been received with healthy 
skepticism 1s that they appear to be so content-free, on the one hand, and 
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methodology -driven, on the other. The history of assessment and o." 
evaluation, as in any new mode of inquiry, is replete with "new" models 
that propound a particular methodological view of the world. A good deal 
of the discredit done to evaluation has occurred with the support and 
consent of its most famous practitioners, who advocated one or another 
highly quantitative design and analysis method as the preferred mode for 
solving all evaluation problems (see Baker, 1983 for a list). 

Obviously, an analytical approach to evaluation design should be 
driven by what information is required by whom by when, by the credibility 
needed by the information analysts to do their job, and most Importantly by 
the nature o* the project or activity under review (Cronbach, 1580). Such 
precepts would suggest an eclectic approach, mixing journalistic, 
documentary, and effectiveness information as appropriate. 

The information must provide policy feedback to the supporting 
agencies . This feature assumes that the funding source 1s either a 
contracting agency or an 1n-house manager. What kinds of policy feedback 
are appropriate? That depends in part on the nature of the formative 
evaluation team seltctea. Clearly, Issues of project management might be a 
necessary concern. However, 1t is more likely that the substance to which 
the technology 1s directed, Instruction, 1s a more useful area for 
feedback. At minimum, the formative evaluators should attend to the 
fidelity of the process by the project to the project's stated goals and 
procedures and to the kinds of contractual, monitoring, and other oversight 
arrangements that might be useful in the future. Furthermore, the 
evaluation report can consider specifically the features or tasks that 
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might be Included 1n the specification of future activities of the sort 
evaluated. 

The tension of providing such Information 1n a way that does not undo 
either the project activities under study or the receptivity of future 
projects to evaluation cannot be Ignored. A fine Hne needs to be walked, 
keeping track of both the professional ethics applicable to contracting 
agency relationships I.e., (telling the truth) and to maintaining positive 
connections to the target R&D communities. 

The Information must provide timely and useful alternatives for 
the formative evaluation of the project(s) under study . This platitude 
takes serious effort to Implement. It depends 1n no small measure 1n 
being Informed accurately and Intimately with the state of development cf 
the project; and 1n the evaluation staff's sensitivity to the form as well 
as the substance of findings that might be useful to the project staff. 
Thi r reoulrement also depends strongly on the level or stage of 
development of the technology activity. Early on, certain suggestions can 
be made and have potentially large eff^ctj. However, early on, the 
evidentiary base of such recommendations 1s likely to be weak. Later on, 
good evidence of project benefits and wtiiknesse- can be more fully drawn; 
however, modification of the technology may be considerably less likely, 
and may cost more. 

Thus, the model addresses macro or executive features of the 
development process rather than micro (or Instructional) characteristics. 
Effectiveness data, based on careful assessment of an appropriate range of 
outcomes, constitute the critical feature of tn1s model, for good 
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management and good documentation have little Importance when the question 
of "does 1t work?" 1s not well treated. We look toward a future 1n which 
such models will be routinely used and rational design and evaluation 
activities will actually drive Instructional development, Instead of 
seemingly evaporating following the approval of a new useful role, more 
realistic and practical than ever. It remains for the field to decide if 
1t wishes to Implement them, and how seriously. 

Summary 

W<» have tried to present 1n this chapter a discussion of outc™e 
assessment that puts Into context how measurement has evolved to Its 
present state. We have attempted to detail the background of alternative 
viewpoints so that the reader can make Informed professional decisions. We 
have also attempted to keep our eye on the ball of Instruction, and urge 
those Interested 1n outcome assessment not to get diverted by the 
intriguing, but occasionally Irrelevant technical debates that suffuse the 
field of psychome tries. Good assessment depends more on hard thinking and 
good analysis than empirical solutions. It 1s for this reuson, we advocate 
the use of criterion referenced measurement for the assessment of 
instructional technology outcomes, with the caveat that such measurement 1s 
difficult and must proceed beyond the often mindless way 1t 1s Implemented 
at present. 

Last, we believe that evaluation of technology outcomes 1s different 
from much of Instructional assessment and that special attention to 
attributes of the assessment model are required. 
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